Booklet

An experimental new standalone, extendable parser-analyzer engine for Lookbook.

The aim is for Booklet to eventually replace the existing file parsing/analyzing code in Lookbook and thus provide a more robust foundation for future releases to build upon.

Additionally, Booklet is being developed as a standalone gem in part so that it can also be used as a springboard for building additional tools (CLIs, MCP servers etc) to help Lookbook integrate more seamlessly into a variety of Rails frontend development workflows.

• Use common, tried-and-tested parser/transformer implementation patterns over custom workflows where possible.
• Implement the analyzing process as a series of small, incremental steps to aid comprehension and testabilty.
• Be backwards compatable with existing Lookbook parser behaviour as surfaced in the app (although underlying APIs and conceptual definitions may differ).
• Make it straightforward to customise and extend the processing pipeline using plugins/middleware.
• Minimise any 'special-casing' of core functionality over that provided by third party extensions.
• Do not have any dependency on Rails (outside of conveniences provided by the ActiveSupport gem).

Booklet is a brand new project and is not yet ready for public use.

The issues list contains an (incomplete) set of work that is planned before an initial beta release will be made available.

Issues specifically related to achieving compatability with Lookbook's current parser output are tagged with the compatability label and the Lookbook compatability project board has been set up to track progress on this metric.

Booklet generates a traversable tree of entity node objects from an input directory of files, via a number of intermediate steps.

Trees can contain a number of different node types. These include folder nodes, file-based entity nodes and entity child nodes that represent the contents of key resource types.

The hierarchy of the nodes in the tree broadly reflects the grouping of input files into folders and subfolders within the root directory as well as parent-child entity-content relationships where present.

All tree mutations and transformations are performed by 'double dispatch'-style node visitors.

Booklet breaks up the processing of files into four steps:

1. File tree creation
2. File tree mutation
3. File tree → entity tree transformation
4. Entity tree mutation

In the first step a tree of generic file and directory nodes is constructed by recursively scanning the root directory and adding a node for every file and folder found. This is handled by the FilesystemLoader visitor.

file_tree = DirectoryNode.from("test/fixtures/demo").accept(FilesystemLoader.new)

ASCII tree visualisation generated using the AsciiTreeRenderer visitor

 [DirectoryNode] demo
 ├── [DirectoryNode] docs
 │   ├── [FileNode] _tmp_notes.txt
 │   ├── [FileNode] banner.png
 │   ├── [FileNode] overview.md
 │   ├── [FileNode] resources.md
 │   └── [DirectoryNode] usage
 │       ├── [FileNode] getting_started.md
 │       ├── [FileNode] installation.md
 │       └── [FileNode] screenshot.svg
 └── [DirectoryNode] view_specs
     ├── [DirectoryNode] elements
     │   ├── [FileNode] button_component_spec.rb
     │   └── [FileNode] card_component_spec.rb
     ├── [FileNode] helpers_spec.rb
     └── [DirectoryNode] layouts
         ├── [FileNode] article_spec.rb
         └── [FileNode] landing_page_spec.rb

This is an optional step where additional node visitors can be provided to mutate the file tree before it is handed off to the entity transformer.

For example you might want to create a custom Visitor class that removes all files with names beginning with an underscore (such as _tmp_notes.txt in the example above) to clean up the tree before the entity transformer is run.

file_tree.accept(SomeCustomFileTreeProcessor.new)

In this step the EntityTransformer visitor is applied to the raw file tree. The transformer visits each of the generic file/directory nodes in the file tree and converts all 'recognized' file types to their corresponding Lookbook entity node type.

For example, files with .md extensions are transformed into DocumentNode instances, whilst component preview class files (names ending in _preview.rb) are transformed into SpecNode instances.

entity_tree = file_tree.accept(EntityTransformer.new)

 [FolderNode] Demo
 ├── [FolderNode] Docs
 │   ├── [AnonNode] _tmp_notes.txt
 │   ├── [AssetNode] banner.png
 │   ├── [DocumentNode] Overview
 │   ├── [DocumentNode] Resources
 │   └── [FolderNode] Usage
 │       ├── [DocumentNode] Getting Started
 │       ├── [DocumentNode] Installation
 │       └── [AssetNode] screenshot.svg
 └── [FolderNode] Previews
     ├── [FolderNode] Elements
     │   ├── [SpecNode] Button Component Preview
     │   └── [SpecNode] Card Component Preview
     ├── [SpecNode] Helpers Preview
     └── [FolderNode] Layouts
         ├── [SpecNode] Article Preview
         └── [SpecNode] Landing Page Preview

This final step is where enity node visitors can be applied to perform tasks such as parsing file contents and generally 'building out' the skeleton entity node objects created in the previous step.

By default Booklet will apply the PreviewClassParser and FrontmatterExtractor visitors at this stage.

entity_tree
  .accept(PreviewClassParser.new)
  .accept(FrontMatterExtractor.new)

• The PreviewClassParser visitor uses the YARD parser to extract annotations data from preview class files and creates and appends corresponding ScenarioNode and ProseNode children to the appropriate SpecNode instance.
• The FrontmatterExtractor visitor (not yet implemented!) extracts YAML-formatted 'frontmatter' from the contents of markdown files and updates the related DocumentNode instances with the parsed data.

Additional entity node vistors can be applied here as needed to make changes to the entity tree nodes before the finalised entity tree is returned for use by the calling code.

Note that the docs branch has been omitted for brevity.

...
└── [FolderNode] Previews
    ├── [FolderNode] Elements
    │   ├── [SpecNode] Button Component Preview
    │   │   ├── [ScenarioNode] Default
    │   │   ├── [ScenarioNode] Secondary
    │   │   └── [ScenarioNode] Danger
    │   └── [SpecNode] Card Component Preview
    │       ├── [ScenarioNode] No Title
    │       └── [ScenarioNode] With Title
    ├── [SpecNode] Helpers Preview
    │   ├── [ScenarioNode] Blah Generator
    │   └── [ScenarioNode] Char Tag Wrapper
    └── [FolderNode] Layouts
        ├── [SpecNode] Article Preview
        │   └── [ScenarioNode] Default
        └── [SpecNode] Landing Page Preview
            └── [ScenarioNode] Default

Booklet is both a command line tool and a library.

Add Booklet to your Gemfile:

gem "lookbooklet"

After running bundle install you can then make use of the Booklet API in your codebase:

require "lookbooklet"

result = Booklet.analyze("path/to/root/directory")

To use the booklet CLI you can install the gem globally:

gem install lookbooklet

You can then view the available booklet CLI commands using:

booklet -h

Details coming soon.

Booklet uses Minitest for its test framework.

Run the tests:

bin/test

Marco Roth's fantastic work on Herb (and my subsequent deep-dive into the world of ASTs) was been instrumental in sparking the initial idea for Booklet and for shaping its approach.

Booklet's double-dispatch style node visitor base class is based on the very nice BasicVisitor class from the Refract gem.

In addition much of the original incarnation of Booklet's Node class was based on code adapted from the excellent RubyTree gem.