Makiri
Makiri is a Ruby library for parsing and querying HTML and XML documents.
It parses HTML with a Rust port of Lexbor's HTML5 parser, and has its own CSS selector matcher, XPath 1.0 engine and XML 1.0 parser. Makiri does not depend on libxml2, and links no C library at all.
Warning
Status: early release. APIs and behavior may change before v1.0.
What / Why
Makiri is one Rust extension: an HTML5 parser ported from Lexbor, CSS selectors, an XPath 1.0 engine and an XML 1.0 parser, with no libxml2 dependency.
- HTML5 parsing
- A Rust port of Lexbor's tokenizer and tree builder, into a compact DOM of Makiri's own. It is held to the vendored Lexbor (which the gem does not ship) and to the html5lib tests.
- CSS selectors
- Selectors 4, including
:is/:where/:not/:hasand:nth-child(... of S), matched by one engine for HTML and XML and checked against Chromium, Firefox and WebKit'squerySelectorAll.
- Selectors 4, including
- Native XPath 1.0 engine
- XPath is parsed and evaluated by Makiri's own engine, written from scratch.
- Makiri does not depend on libxml2 for parsing, DOM representation, or XPath evaluation.
- Native XML 1.0 parser
- A strict, non-validating, fail-closed parser with its own node arena (not the HTML DOM), queried through the same native XPath engine, with in-place tree edits (attributes, content, remove).
- Conformance is held by the W3C XML Conformance Test Suite, an XPath differential, and property-based testing vs Nokogiri (see below).
- Bounded, fail-closed execution
- XPath evaluation is bounded by per-evaluation limits on work, memory, and recursion.
- HTML parsing bounds the tree depth (
max_tree_depth:, default 400) and the size of the document it builds against its input (max_amplification:, default 256 times, never less than 64 MiB). - Ownership and borrowing are kept explicit across layers, with owned/borrowed string types and verified text at engine boundaries.
- Programmatic invalid input, limit violations, allocation failures, and unsupported constructs fail closed instead of producing partial or silently truncated results.
Usage
require "makiri"
doc = Makiri::HTML(<<~HTML)
<html><body>
<div id="main" class="container">
<p class="lead">Hello</p>
<a href="/a">one</a>
<a href="/b">two</a>
</div>
</body></html>
HTML
# CSS selectors
doc.css("a").map { |a| a["href"] } # => ["/a", "/b"]
doc.at_css("p.lead").text # => "Hello"
# XPath 1.0 (native engine - no libxml2)
doc.xpath("//a").length # => 2
doc.xpath("count(//a)") # => 2.0
doc.at_xpath('//*[@id="main"]/p').text # => "Hello"
# Attributes and navigation
link = doc.at_css("a")
link["href"] # => "/a"
link.parent.name # => "div"
# Source location (the line of the element's start tag)
doc.at_css("p").line # => 3
# Bytes are decoded as a browser decodes them: BOM, <meta charset>, else
# UTF-8 or a detected legacy encoding (or say which with encoding:)
Makiri::HTML(File.binread("legacy.html")).encoding # => "Shift_JIS"
Makiri::HTML(bytes, encoding: "euc-jp")
# Nesting deeper than 400 elements raises Makiri::Error (Nokogiri's default);
# max_tree_depth: raises the limit, and a negative value disables it
Makiri::HTML(deep_html, max_tree_depth: 2000)
# A short input cannot build an enormous document: more than 256 times the
# input (and 64 MiB) raises Makiri::Error; max_amplification: changes the ratio,
# and a negative value disables it
Makiri::HTML(html, max_amplification: 1024)
# Serialization
doc.at_css("#main").to_html # => "<div id=\"main\" ...>...</div>"
doc.at_css("#main").inner_html # => "\n <p class=\"lead\">Hello</p>\n..."XPathContext (namespaces and variables)
ctx = Makiri::XPathContext.new(doc)
ctx.register_variable("cls", "lead")
ctx.evaluate('//p[@class=$cls]').first.text # => "Hello"XML (with in-place editing)
doc = Makiri::XML(<<~XML)
<feed xmlns="http://www.w3.org/2005/Atom">
<entry><title>Hello</title></entry>
<entry><title>World</title></entry>
</feed>
XML
# Namespace matching is strict, so a default namespace needs a registered prefix.
ns = { "a" => "http://www.w3.org/2005/Atom" }
doc.xpath("//entry").length # => 0 (default namespace)
doc.xpath("//a:entry", ns).length # => 2
doc.at_xpath("//a:entry/a:title", ns).text # => "Hello"
# Or reuse a context (caches registrations + compiled expressions):
ctx = Makiri::XPathContext.new(doc.root)
ctx.register_namespace("a", "http://www.w3.org/2005/Atom")
ctx.evaluate("//a:entry").length # => 2
el = doc.at_xpath("//a:entry", ns)
el.local_name # => "entry"
el.namespace_uri # => "http://www.w3.org/2005/Atom"
# CSS selectors work too, as in a browser: a bare type selector matches its
# name in any namespace, so this just works.
doc.css("entry").length # => 2
doc.css("a|entry", ns).length # => 2 (a prefix the Hash binds)
doc.css("feed > entry").map { |e| e.at_css("title").text } # => ["Hello", "World"]
# Serialize back to XML
doc.to_xml # => "<?xml version=\"1.0\"?>\n<feed ...>...</feed>\n"
# A node below the root serializes self-contained: no XML declaration, but the
# namespace declarations its subtree needs, so the output re-parses the same.
doc.at_xpath("//a:entry", ns).to_xml # => "<entry xmlns=\"http://www.w3.org/2005/Atom\"><title>Hello</title></entry>"
doc.to_xml(pretty: true) # indented, element-only content
# DOCTYPE is recognized but the DTD is not processed (no entities, no I/O):
dtd = Makiri::XML(%(<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0//EN" "x.dtd"><html/>))
.internal_subset
dtd.name # => "html"
dtd.external_id # => "-//W3C//DTD XHTML 1.0//EN" (alias: #public_id)
dtd.system_id # => "x.dtd"The tree supports in-place mutation.
doc = Makiri::XML(%(<feed xmlns:dc="urn:dc"><entry id="1">Hi</entry><draft/></feed>))
e = doc.at_xpath("//entry")
e["id"] = "9" # add or replace an attribute (value escaped on output)
e["dc:k"] = "v" # a prefixed name resolves against the in-scope xmlns
e.content = "Bye" # replace an element's children with text
e.delete("id") # remove an attribute
doc.at_xpath("//draft").remove
doc.root.to_xml # => "<feed xmlns:dc=\"urn:dc\"><entry dc:k=\"v\">Bye</entry></feed>"XML subtrees can be built using Document#create_element and other node factory methods,
then inserted into a document with #add_child, #before, #after, or #replace.
When a node is created, it inherits its namespace from the context where it is first inserted; if it already has a namespace, that namespace is preserved.
doc = Makiri::XML(%(<feed xmlns="urn:a" xmlns:dc="urn:dc"/>))
entry = doc.create_element("entry")
entry["dc:id"] = "42" # prefixed attr resolves on insertion
entry.add_child(doc.create_element("title", "Hello"))
doc.root.add_child(entry)
doc.to_xml # => "...<entry dc:id=\"42\"><title>Hello</title></entry>..."Makiri::XML::Builder is the Nokogiri-compatible DSL over those factories.
builder = Makiri::XML::Builder.new do |xml|
xml.feed("xmlns" => "http://www.w3.org/2005/Atom", "xmlns:dc" => "urn:dc") do
xml.title("Example Feed")
xml.entry("dc:id" => "1") do
xml.title("First")
xml.summary { xml.cdata("raw <b>html</b>") }
end
end
end
builder.to_xml # the whole document (with XML declaration)
builder.doc # the Makiri::XML::Document being builtXML parsing is bounded by an arena memory limit, 256 MiB by default,
and unusually large documents can raise it with max_bytes:.
Makiri::XML(huge_xml, max_bytes: 512 * 1024 * 1024) # also Makiri::XML::Document.parse(..., max_bytes:)The limit covers the document's whole life, not only the parse: nodes and
values made by editing it count too, and a replaced value or a removed node
keeps its bytes until the document is gone. A document edited for a long time
can be compacted with dup, which copies only what is in it now, under the
same limit.
Node identity
Navigating to the same node always gives the same Ruby object, for as long as
its document is alive. The document keeps every wrapper it has handed out, so
instance variables, singleton methods and freeze set on a node survive
garbage collection and are there the next time the node is reached.
That makes a node a safe place to keep per-node state, such as a wrapper object of your own:
el = doc.at_css("p")
el.instance_variable_set(:@wrapper, MyElement.new(el))
doc.at_css("p").instance_variable_get(:@wrapper) # => the same MyElementA node created by clone_node or import_node is a new node with no state.
pointer_id is unique only among live nodes: once a document is freed, a node
of another document may reuse its value. So key a pointer-based cache per
document, not across documents.
Non-goals (v1.0)
- XSLT, DTD / Schema / RelaxNG validation, XPointer, XInclude.
- Streaming / SAX parsing.
- Drop-in replacement for every Nokogiri method. Makiri covers the common
HTML-scraping and manipulation surface. Deliberately not provided:
- XHTML serialization variants (
to_xhtml,write_xml_to);#to_xmlis supported - XML/DTD construction (
create_internal_subset,external_subset) - namespace mutation (
add_namespace_definition); read introspection (#namespace,#namespace_definitions,#namespaces,#collect_namespaces) is supported onMakiri::XMLnodes - Nokogiri internals (
decorate,slop!,validate).
- XHTML serialization variants (
Differences from Nokogiri
Makiri targets a Nokogiri-compatible API, but a few behaviours differ - XPath name matching, XML namespaces, CSS extensions, serialization and text input. See NOKOGIRI_DIFFERENCES.md.
Conformance
The XPath engine and XML parser are original code, so their correctness is held by
differential and standards harnesses in spec/conformance/.
The HTML XPath and CSS suites are differentials against Nokogiri::HTML5
(Gumbo / WHATWG, never libxml2's non-conformant HTML4 parser): both sides parse
HTML5, so the DOM is isomorphic and results are compared node-for-node. HTML
parsing itself is checked against the WHATWG html5lib-tests corpus, and
XPath-over-HTML semantics additionally against browsers via a WPT port.
HTML parsing and CSS selectors are also run in Chromium, Firefox and WebKit,
which are the reference where they and Nokogiri disagree.
See also spec/conformance/README.md.
| Suite | Input | Oracle |
rake task |
|---|---|---|---|
| HTML parsing | HTML | WHATWG html5lib-tests (expected-tree corpus) | conformance:html5 |
| HTML parsing | HTML | Chromium, Firefox, WebKit (Playwright) | conformance:browsers |
| XPath 1.0 | HTML |
Nokogiri::HTML5 (libxml2 XPath) — differential |
conformance:xpath |
| XPath over HTML | HTML | browsers (WPT domxpath, hand-ported; runs under rake spec) |
— |
| CSS selectors | HTML |
Nokogiri::HTML5#css — differential |
conformance:css |
| Well-formedness | XML | W3C XML Conformance Test Suite | conformance:xmlconf |
| XPath 1.0 | XML |
Nokogiri::XML — differential |
conformance:xpath_xml |
| Parsed tree (property-based) | XML |
Nokogiri::XML — differential |
conformance:xml_pbt |
| CSS selectors | XML |
Nokogiri::XML — differential |
conformance:css_xml |
| CSS selectors | HTML, XML | Chromium, Firefox, WebKit querySelectorAll (Playwright) |
conformance:css_browsers |
| XML Builder | Ruby DSL |
Nokogiri::XML::Builder — differential |
conformance:builder |
Requirements
- CRuby 3.3 or newer.
- Rust 1.84 or newer with
cargo- the extension is a Rust crate. (Binary gems are published for the common platforms, which need no toolchain.) - libclang (clang's development library): rb-sys runs bindgen at build time and reads Ruby's headers through it.
Development also needs CMake and a C toolchain, to build the vendored Lexbor
that the Rust port is tested against (rake lexbor).
Build (development)
git submodule update --init --recursive
bundle install
bundle exec rake compile
bundle exec rake spec