A Scripter's Notes ❚
Emacs, scripting and anything text oriented.

Defining tomelr – A library for converting Lisp expressions to TOML

— Kaushal Modi

Creating a specification for an Emacs-Lisp library to convert Lisp data expressions into easy-to-read TOML strings.

Table of Contents

ox-hugo has some custom code that generates TOML  I ❤️ TOML. As the makers of this config format put it.. “it’s a format for humans”! — No need to deal with indentations, weird syntax for multi-line strings, no prohibition on adding comments to the config, or dealing with careful placement of commas and braces. for Hugo front-matter, based on the Org keywords and other meta-data that the user sets in their Org file. But this TOML generation code is not generic enough for any TOML object type.

As I am writing up a definition on how to export :LOGBOOK: drawers, I felt it’s a good time to polish the whole Lisp data → TOML conversion code, and may be package that into a separate library.

        It’s kind of an ambitious project — I am calling it tom​el​r ✨

It’s a big undertaking to create a generic library for this kind of data format conversion. But even before I start coding or think if I can complete this project, I need to spec  I am not sure if it’s a widely used verb, but to spec means to write a specification for something. it. I need to understand early-on how the S-exp (Symbolic lisp expression) would needed to look for each kind of generated TOML object — scalars, lists, lists of lists, maps, lists of maps, etc.

Using json-encode as reference #

The aim of the tomelr library is to take some (lisp data) and convert that TOML. But I did not want to invent my own lisp data convention for this! So I decided to stick with the lisp expression conventions understood by the json-encode function from the Emacs core library json.el.

Credit for the json.el idea goes to this tweet by Piers Cawley:

I’d suggest that  By “that”, he’s referring to adding support for exporting front-matter to JSON in ox-hugo. , since emacs has built in JSON support these days, you don’t really have to worry about the commas and braces, just build the s-exp you want and export, but it’s you that’s writing the code and I’m just delighted that it exists.

Thank you for your efforts.

I am not sold on adding support of yet another front-matter format to ox-hugo. I might not use json.el for that, but it definitely helped me a lot with coming up with this library’s spec 😆.

Mapping scalar data to TOML #

Figuring out the Lisp representation for scalar (plain key-value pairs) TOML objects was easy. json.el helped figure out how to deal with nil and false values.

Table 1: Mapping of scalar Lisp data to TOML
Lisp S-expTOML
'((int_key . 123))int_key = 123
'((float_key . 1.23))float_key = 1.23
'((date_key . 2022-04-27))date_key = 2022-04-27
'((bool_key . t))bool_key = true
'((any_key . nil))(key removed in TOML)
'((bool_key . :false))bool_key = false

About :false #

json.el defines a variable json-false that’s set to the value :json-false. This is because in JSON, the null value is different from the boolean false value.

  • nil in Lisp → null in JSON
  • :json-false in Lisp → false in JSON

Inspired by that decision of json.el, I am thinking of using :false as the special value that will set the equivalent TOML value to boolean false.

TOML does not define a null value, but if the Lisp value is nil, that key will simply not be translated to TOML.

Mapping lists to TOML #

Mapping lists was simple.. because in Lisp, a list value of course looks like '((foo . (1 2 3 4 5))) 😃.

Table 2: Mapping of list Lisp data to TOML
Lisp S-expTOML
'((list_key1 . (1 2 3)))list_key1 = [1, 2, 3]
'((list_key2 . ("a" "b" "c")))list_key2 = ["a", "b", "c"]

Mapping lists of lists to TOML #

Table 3: Mapping of list of list Lisp data to TOML
Lisp S-expTOML
'((lol_key . [(1 2) (3 4)]))lol_key = [ [1, 2], [3, 4] ]

I was going to use '((lol_key . ((1 2) (3 4)))) as the reference Lisp expression for lol_key = [ [1, 2], [3, 4] ]. But I found out that json-encode throws an error if you pass it that expression! I don’t understand the reason for that error, and so I have asked for help on the help-gnu-emacs mailing list.

But while that question gets resolved, I wanted to move forward with the spec definition. After some trial-and-error and reverse-engineering json.el  I knew how I wanted TOML to look. So I used an online JSON/TOML converter to convert that TOML snippet to JSON, and then used json-read to convert JSON to Lisp expression. , I learned that list of list data needs to be represented using a Vector type, and so '((lol_key . [(1 2) (3 4)])) would be the correct expression – Notice the use of square brackets instead of parentheses for the outer vector.

Mapping other object types #

Once I figured out how to map the above data types, mapping Lisp data to TOML Tables aka dictionaries and arrays of Tables was a breeze  Of course, the breeze is referring to the ease of writing the spec for these 😆. Implementation-wise, the tables, arrays of tables, and the especially nested variants of those are going to be the most challenging. .

Closing #

It was fun coming up with an initial draft of the specification for this library. My next steps would be to gradually add TOML generator functions (as I find time) to this library, along with ERT tests! Eventually, I will remove the existing TOML generation code from ox-hugo and depend on this library.

Getting back to my plan for exporting :LOGBOOK: drawers in ox-hugo, based on this spec, I will need to construct this date in Emacs-Lisp:

(org_logbook . (((timestamp . 2022-04-08T14:53:00-04:00)
                 (note . "This note addition prompt shows up on typing the `C-c C-z` binding.\nSee [org#Drawers](https://www.gnu.org/software/emacs/manual/html_mono/org.html#Drawers)."))
                ((timestamp . 2018-09-06T11:45:00-04:00)
                 (note . "Another note **bold** _italics_."))
                ((timestamp . 2018-09-06T11:37:00-04:00)
                 (note . "A note `mono`."))))
Code Snippet 1: Example data from a :LOGBOOK: drawer in Lisp format

.. will translate to this in TOML:

[[org_logbook]]
  timestamp = 2022-04-08T14:53:00-04:00
  note = """This note addition prompt shows up on typing the `C-c C-z` binding.
See [org#Drawers](https://www.gnu.org/software/emacs/manual/html_mono/org.html#Drawers)."""
[[org_logbook]]
  timestamp = 2018-09-06T11:45:00-04:00
  note = """Another note **bold** _italics_."""
[[org_logbook]]
  timestamp = 2018-09-06T11:37:00-04:00
  note = """A note `mono`."""
Code Snippet 2: Same example data from the :LOGBOOK: drawer translated to TOML format

Check out the below link where I have documented the equivalent expressions between Lisp, TOML and JSON for all the object types.

👉 tom​el​r Specification