Project

livetext

0.01
A long-lived project that still receives updates
A smart text processor extensible in Ruby
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies
 Project Readme

Livetext: A smart processor for text

This README is currently mangled. Fixes coming soon!

Livetext is simply a tool for transforming text from one format into another. The source file has commands embedded in it, and the output is dependent on those commands.

Why is this special? It's very flexible, very extensible, and it's extensible in Ruby.

Why Livetext?

Livetext grew out of several motivations. One was a desire for a markup language that would permit me to write articles (and even books) in my own way and on my own terms. I've done this more than once (and I know others who have, as well).

I liked Softcover, but I found it to be very complex. I never liked Markdown much -- I find it very dumb, and it's not extensible at all. (In fairness to Markdown, it does serve a different purpose in everyday life.)

I wanted something that had the basic functionality of all my ad hoc solutions but allowed extensions. Then my old solutions would be like subsets of the new format. This was a generalization similar to the way we began several years ago to view HTML as a subset of XML.

What is Livetext really?

Here goes:

  • It's a text transformer
  • It's Ruby-based (later on, more language agnostic)
  • It's (potentially) agnostic about output format
  • It's designed to be flexible, extensible, and easy
  • It's designed to be "plugin" oriented
  • It's like an old-fashioned text formatter (but extensible)
  • It's like a macro processor (but not)
  • It's like markdown and others (but not)
  • It's like erb or HAML (but not)
  • It's powerful but not too dangerous
  • It's not necesarily a markdown replacement
  • It's definitely not a softcover replacement
  • It could possibly augment markdown, softcover, others

How does it work?

A Livetext file is simply a text file which may have commands interspersed. A command is simply a period followed by a name and optional parameters (at the beginning of a line).

The period will be configurable later if you want to use another character. The names are (for now) actual Ruby method names, so names such as to_s and inspect are currently not allowed.

At present, I am mostly emitting "dumb HTML" or Markdown as output. In theory, you can write code (or use someone else's) to manipulate text in any way and output any format. Technically, you could even emit PDF, PNG, or SVG formats.

It's possible to embed comments in the text. Later it will be possible to pass them through to the output in commented form.

The command .end is special, marking the end of a body of text. Some commands may operate on a block of lines rather than just a few parameters. (A text block is like a here-document.) There is no method name corresponding to the .end command.

The file extension I've chosen is .lt3 (though this may change). Note: The source for this README is a .lt3 file which uses its own little ad hoc library (called tutorial.rb). Refer to the repo to see these.

Syntax, comments, and more

At first, my idea was to provide predefined commands and allow user-defined commands (to be distinguished by a leading . or .. marker). So the single and double dots were both legal.

However, my concept at present is that the double dots (currently unused) may be used for subcommmands.

User-defined commands may be added to the standard namespace. There are plans to permit commands beginning with a specified character other than the period (to be stored in their own namespace.

When a leading period is followed by a space, that line is a comment. When it is follwed by a name, that name is typically understood to be a method name. Any remaining text on the line is treated as a parameter list to be accessed by that method. Some methods accept a text block (multiple lines of text terminated by a .end tag).

Boldface and italics

Very commonly we want to format short words or phrases in italics, boldface, or a monospaced (fixed width) font. The Markdown spec provides ways to do this that are fairly intuitive; but I personally don't like them. My own notation works a different way.

First of all, note that these don't work across source lines; they're strictly intra-line. You may need (for example) an italicized phrase that spans across a newline; at present, you'll need a workaround for that.

I find that most short items I want to format are single tokens. Therefore I use a prefixed character in front of such a token: Underscore for italics, asterisk for boldface, and backtick for "code font." The formatting ends when the first blank space is encountered, without any kind of suffixed character.

I also find it's common to want to terminate such a string with some kind of naturally-occurring punctuation mark. If we double the initial delimiter, it will be understood to terminate at the first period, comma, or right parenthesis.

Of course, there are cases where this won't work; a formatted string may contain spaces, or it may exclude characters before the blank space. In this case, we can use an opening bracket after the prefix and a closing bracket at the end of the string.

This means that it can be difficult to include brackets inside a formatted token. The solution is simply to escape with a backslash.

A delimiter character sitting by itself need not be escaped. It will be output as a literal.

A delimiter character that is already inside another string need not be escaped. These cannot be nested (though there is a way to accomplish this using functions).

Most of this is summarized in this example (taken from one of the testcases):

Test: basic_formatting

Input Output
 Here are examples of *boldface
and _italics
and code as well as *[more complex] examples of \_[italicized text] and [code font].

Here are some random punctuation marks: ; # . * _ ` : @ % ^ & $

No need to escape these: * _ `

 Here are examples of boldface
and italics
and code
as well as more complex examples
of italicized text
and code font.

Here are some random punctuation marks: ; # . * _ ` : @ % ^ & $

No need to escape these: * _ `


Standard methods

The module Livetext::Standard contains the set of standard or predefined methods. Their names are essentially the same as the names of the dot-commands, with occasional exceptions. (For example, it is impractical to use the name def as a method name, so the module has a _def method instead.) Here is the current list:


comment Start a comment block
errout Write an error message to STDERR
def Define a new method inline
set Assign values to variables for later interpolation
include Include an outside text file (to be interpreted as Livetext)
mixin Mix this file of Ruby methods into the standard namespace
copy Copy this input file verbatim (no interpretation)
r Pass a single line through without processing
raw Pass this special text block (terminated with __EOF__) directly into output without processing
func Define a function to be invoked inline
say Print a message to the screen
banner Print a "noticeable" message to the screen
quit End processing and exit
nopass Don't pass lines through (just honor commands)
include Read and process another file (typically a .lt3 file)
debug Turn on debugging
nopara Turn off the "blank line implies new paragraph" switch
newpage Start a new output page

Examples from the tests

Here are some tests from the suite. The file name reflects the general purpose of the test.

Test: hello_world

Input Output
 Hello,
world!
 Hello,
world!

Test: comments_ignored_1

Input Output
 . Comments are ignored
abc 123
this is a test
. whether at beginning, middle, or
more stuff
still more stuff
. end of the file
 abc 123
this is a test
more stuff
still more stuff

Test: block_comment

Input Output
 .comment
This is
a comment
.end
abc 123
xyz
.comment
And so is this.
.end

one more time .comment And so is this .end

 abc 123
 xyz
 one
 more
 time

Test: def_method

Input Output
 .backtrace
abc
123
.def foobar     # yet another experimental comment
::STDERR.puts "This is the"
::STDERR.puts "foobar method"
.end
xyz
.foobar     # and still another
xyzzy
123
 abc
123
xyz
xyzzy
123

Test: simple_vars

Input Output
 Just
some text.
.set name=GulliverFoyle, nation=Terra
Hi, there.
$name is my name, and $nation is my nation.
I'm $name, from $nation.
That's all.
 Just
some text.
Hi, there.
GulliverFoyle is my name, and Terra is my nation.
I'm GulliverFoyle, from Terra.
That's all.

Test: simple_include

Input Output
 Here I am
trying to
include
.include simplefile.inc
I hope that
worked.
 Here I am
trying to
include
a simple
include file.
I hope that
worked.

Test: simple_mixin

Input Output
 Here I am
testing a simple mixin
.mixin simple_mixin
Now call a method:
.hello_world
That's all.
 Here I am
testing a simple mixin
Now call a method:
Hello, world.
That's all.

Test: simple_copy

Input Output
 The copy command
copies any file
without interpretation,
such as:
.copy simplefile.inc
That is all.
 The copy command
copies any file
without interpretation,
such as:
a simple
include file.
That is all.

Test: copy_is_raw

Input Output
 A copy command
does not interpret its input:
.copy rawtext.inc     # another useless comment
That's all.
 A copy command
does not interpret its input:
This is not a comment:
.comment woohoo!
This is not a method:
.no_such_method
That's all.

Test: raw_text_block

Input Output
 .backtrace
This text block will be passed thru
with no interpretation or processing:
.raw
.comment
This isn't a
real comment.
.end  This isn't picked up.

.not_a_method

And this stuff won't be munged: `alpha _beta *gamma

Or these: `(alpha male) _(beta max) *(gamma rays) __EOF__

 This text block will be passed thru
with no interpretation or processing:
.comment
This isn't a
real comment.
.end  This isn't picked up.

.not_a_method

And this stuff won't be munged: `alpha _beta *gamma

Or these: `(alpha male) _(beta max) *(gamma rays)


Writing custom methods

Suppose you wanted to write a method called chapter that would simply output a chapter number and title with certain heading tags and a horizontal rule following. There is more than one way to do this.

The simplest way is just to define a method inline with the rest of the text. Here's an example.

     .comment
     This example shows how to define
     a simple method "chapter" inline
     .end
   
     . This is also a comment, by the way.
     .def chapter
        params = <i>args</i>
        raise "chapter: expecting at least two args" unless params.size > 1
        num, <b>title</b> = params     # Chapter number + title
        title = title.join(" ")  # Join all words into one string
        text = <<-HTML
        <h3>Chapter #{num}</h3>
        <h2>#{title}</h2>
        <hr>
        HTML
        <i>puts</i> text
     .end
     . Now let's invoke it...
     .chapter 1 Why I Went to the Woods
     It was the best of times, and you can call me Ishmael. The clocks
     were striking thirteen.

What can we see from this example? First of all, notice that the part between .def and .end (the body of the method) really is just Ruby code. The method takes no parameters because parameter passing is handled inside the Livetext engine and the instance variable @_args is initialized to the contents of this array. We usually refer to the @_args array only through the method _args which returns it.

The _args method is also an iterator. If a block is attached, that block will be called for every argument.

We then create a string using these parameters and call it using the _puts method. This really does do a puts call, but it applies it to wherever the output is currently being sent (defaulting to STDOUT).

All the "helper" methods start with an underscore so as to avoid name collisions. These are all stored in the Livetext::UserAPI module (which also has some methods you will never use).

Here is the HTML output of the previous example:

     <h3>Chapter 1</h3>
     <h2>Why I Went to the Woods</h2>
     <hr>
     It was the best of times, and you can call me Ishmael. The clocks
     were striking thirteen.

What are some other helper methods? Here's a list.


_args Returns an array of arguments for the method (or an enumerator for that array)
_data A single "unsplit" string of all arguments in raw form
_body Returns a string (or enumerator) giving access to the text block (preceding .end)
_puts Write a line to output (STDOUT or wherever)
_print Write a line to output (STDOUT or wherever) without a newline
_formatting A function transforming boldface, italics, and monospace (Livetext conventions)
_passthru Feed a line directly into output after transforming and substituting
Note that the last three methods are typically not called in your own code. They could be, but it remains to be seen whether something that advanced is useful.

More examples

Suppose you wanted to take a list of words, more than one per line, and alphabetize them. Let's write a method called alpha for that. This exercise and the next one are implemented in the test suite.

Test: example_alpha

Input Output
 .def alpha
text = _body.to_a.join(" ")
words = text.split.sort
words.each {|w| _out "    #{w}" }
.end
Here is an alphabetized list:

.alpha fishmonger anarchist aardvark glyph gryphon halcyon zymurgy mataeotechny zootrope pareidolia manicotti quark bellicose anamorphic cytology fusillade ectomorph .end

That's all.

 Here is an alphabetized list:

 aardvark
 anamorphic
 anarchist
 bellicose
 cytology
 ectomorph
 fishmonger
 fusillade
 glyph
 gryphon
 halcyon
 manicotti
 mataeotechny
 pareidolia
 quark
 zootrope
 zymurgy

That's all.


I'll let that code stand on its own. Now suppose you wanted to allow columnar output. Let's have the user specify a number of columns (from 1 to 5, defaulting to 1).

Test: example_alpha2

Input Output
 .def alpha
cols = _args.first
cols = "1" if cols == ""
cols = cols.to_i
raise "Columns must be 1-5" unless cols.between?(1,5)
text = _body.join("\n")
text.gsub!(/\n/, " ")
words = text.split.sort
words.each_slice(cols) do |row|
row.each {|w| _out! '%-15s' % w }
_out
end
.end
Here is an alphabetized list:

.alpha 3 fishmonger anarchist aardvark glyph gryphon halcyon zymurgy mataeotechny zootrope pareidolia manicotti quark bellicose anamorphic cytology fusillade ectomorph .end

And that is all.

 Here is an alphabetized list:

aardvark anamorphic anarchist bellicose cytology ectomorph fishmonger fusillade glyph gryphon halcyon manicotti mataeotechny pareidolia quark zootrope zymurgy

And that is all.


What if we wanted to store the code outside the text file? There is more than one way to do this.

Let's assume we have a file called mylib.rb in the same directory as the file we're processing. (Issues such as paths and security have not been addressed yet.) We'll stick the actual Ruby code in here (and nothing else).

   # File: mylib.rb
   
   def alpha
     cols = <i>args.first</i>
     cols = "1" if cols == ""
     cols = cols.to_i
     raise "Columns must be 1-5" unless cols.between?(1,5)
     text = <i>body.join</i>
     text.gsub!(/n/, " ")
     words = text.split.sort
     words.each_slice(cols) do |row| 
       row.each {|w| <i>print</i> '%-15s' % w }
       <i>puts</i> 
     end
   end

Now the .lt3 file can be written this way:

    .mixin mylib
    Here is an alphabetized list:
   
    .alpha 3
    fishmonger anarchist aardvark glyph gryphon
    halcyon zymurgy mataeotechny zootrope
    pareidolia manicotti quark bellicose anamorphic
    cytology fusillade ectomorph
    .end
   
    And that is all.

The output, of course, is the same.

You can define variables in Livetext, defined with .set and referenced with a $$. Later there will be a few predefined variables. Variables are just string values.

Test: simple_vars

Input Output
 Just
some text.
.set name=GulliverFoyle, nation=Terra
Hi, there.
$name is my name, and $nation is my nation.
I'm $name, from $nation.
That's all.
 Just
some text.
Hi, there.
GulliverFoyle is my name, and Terra is my nation.
I'm GulliverFoyle, from Terra.
That's all.

If a variable needs to contain spaces, you can double-quote it.

Test: more_complex_vars

Input Output
 Just some more text.
.set bday="May 31", date="5/31"
My birthday is $bday, so they tell me.
That's $date if you're American.
That's all.
 Just some more text.
My birthday is May 31, so they tell me.
That's 5/31 if you're American.
That's all.

Livetext permits user-defined functions (as well as defining a few predefined ones). Call a function with $$ and (if applicable) pass a single string parameter between brackets.

Test: functions

Input Output
 Testing out
some functions
here...

.func myfunc "Eureka!" .end

I am calling $$myfunc here... Let's see what happens.

Functions can take a parameter in brackets (which it can parse itself as needed):

.func mean list = param.split(",").map(&:to_f) sum = list.inject(0, :+) avg = sum / list.size.to_f avg.to_s .end

The result is $$mean[1,2,3,4,5,6,7] as I see it.

Some functions like date and time are predefined.

 Testing out
 some functions
 here...
 

I am calling Eureka! here... Let's see what happens.

Functions can take a parameter in brackets (which it can parse itself as needed):

The result is 4.0 as I see it.

Some functions like date and time are predefined.


There is an important feature that has not yet been implemented (the require method). Like Ruby's require, it will grab Ruby code and load it; however, unlike mixin, it will load it into a customized object and associate a new sigil with it. So for example, the command .foobar would refer to a method in the Livetext::Standard class (whether predefined or user-defined). If we did a require on a file and associated the sigil # with it, then #foobar would be a method on that new custom object. I plan to implement this later.

Issues, open questions, and to-do items

This list is not prioritized yet.

  • Add versioning information
  • Clean up code structure
  • Add RDoc
  • Think about command line executable
  • Write as pure library in addition to executable
  • Package as gem
  • Document: require include copy mixin errout and others
  • Need much better error checking and corresponding tests
  • Worry about nesting of elements (probably mostly disallow)
  • Think about UTF-8
  • Document API fully
  • Add _raw_args and let _args honor quotes
  • Support quotes in .set values
  • Support "namespaced" variables (`[.set code.font="whatever"])
  • Support functions (``[func is undefined]$c)
  • Support function namespacing
  • Create predefined variables (e.g., [source_file is undefined]$, $_line])
  • Create predefined functions (e.g., $_date)
  • More support for markdown
  • Allow turning on/off: formatting, variable interpolation, function interpolation?
  • .require with file and sigil parameters
  • Investigate "common intermediate format" - output renderers all read it
  • Comments passed through (e.g. as HTML comments)
  • .run to execute arbitrary Ruby code inline?
  • Concept of .proc (guaranteed to return no value, produce no output)?
  • Exceptions??
  • Ruby [AFE is undefined]$ levels?
  • Warn when overriding existing names?
  • Think about passing data in (erb replacement)
  • ]Allow custom ending tag on raw method
  • Ignore first blank line after `[.end? (and after raw-tag?)
  • Allow/encourage custom passthru method?
  • Must have sane support for CSS
  • Support for Pygments and/or other code processors
  • Support for gists? arbitrary links? other remote resouces?
  • Small libraries for special purposes (books? special Softcover support? blogs? PDF? RMagick?)
  • Experiment with idea of special libraries having pluggable output formats (via Ruby mixin?)
  • Imagining a lib that can run/test code fragments as part of document generation
  • Create vim (emacs?) syntax files
  • Someday: Support other languages (Elixir, Python, ...)
  • .pry method?
  • .irb method?
  • Other debugging features
  • Feature to "break" to EOF?
  • .meth? method ending in ? takes a block that may be processed or thrown away (`.else perhaps?)
  • .dump to dump all variables and their values
  • .if and .else?
  • Make any/all delimiters configurable
  • HTML helper? (in their own library?)