In this post, I introduce a little library I created for
have a robust mechanism for generating TOML from any Lisp expression.
In my previous post Defining tomelr, I started toying with the idea of creating a library that would help convert any Lisp expression to a TOML config, and I share my vision (specification) of what this library would look like.
I wasn’t even sure if I would be able to make the tomelr library
feature-complete at least to the extent of what
ox-hugo was already
doing! But to my surprise, the library development snowballed to a
completion much earlier than I thought, and additionally it helped fix
some inconsistencies that the older TOML generation code had in
In this post, I start by (i) giving a broad overview of how the
tomelr happened, then (ii) briefly describe how it
got integrated into
ox-hugo, and finally (iii) how the use of this
library will unblock the path to addition of some cool features to
My flow for creating this library #
- Write the spec for the library.
- List all the formats of Lisp data I would expect it to process.
- List the corresponding TOML data I would expect it to generate.
- Ensure that I am not inventing my own lisp syntax by confirming
that the expected TOML output matches the JSON generated from
that same lisp form (using the Emacs built-in
- That helped me write the tests first! – Test Driven Development (TDD).
- I started with writing tests for TOML booleans and then
implementing that (because that was the simplest and easiest). Of
course, I used ert for this!
erthelped me quickly create small modular tests Here’s the ert test for booleans as an example. and efficiently iterate through modifications in the library code until I got the tests to pass.
- Once that got working, I set up a continuous integration system using GitHub Actions (GHA). I used GHA because I host my library on GitHub. Also I already have a tried and tested setup that I could get up and going in a matter of few seconds. In general, this concept would apply to any Continuous Integration system. The CI setup step should come early in the development of any project so that incremental feature additions don’t start breaking previously added features 😃.
- The library development just snowballed after this point .. added support for integers, floats, regular strings, multi-line strings, arrays, TOML tables, arrays of TOML tables. By this time, the library was about 80% finished.
- Then came the difficult part .. stabilizing the library to support all the varieties of Lisp data I can think of. I must have put in double the time spent so far to finish the remaining 20% of the planned features for this library 👉 tomelr test suite
- Once I had the test suite complete and passing, it was time to do
some code cleanup:
- Remove duplicate code and break them off into smaller helper functions.
- See if the function defined in this library is already defined
somewhere else (in this case, I was able to use
- Proof read the code.
- Proof read the docstrings and run
M-x checkdocto fix their formatting.
- Ensure that the code compiles without any warnings.
- Remove unnecessary customization options and case statements from the library (while continuously ensuring that the ert tests still pass).
Adapting the library to fit
After polishing the library by its stand-alone testing, I decided to
use it with
ox-hugo and see how the test suite in that repo fared.
Of course I saw that a lot of tests failed now 😁.
The main issue was that
tomelr was constructing multi-line strings
such that the spaces translated exactly from Lisp data to TOML. So
(tomelr-encode '((foo . "line1\nline2"))) would generate:
foo = """ line1 line2"""
ox-hugo expected the same TOML to look like:
foo = """ line1 line2 """
I had intentionally decided for
ox-hugo to have this latter format
for multi-line strings because (i) it made it more readable with the
triple-quotes out of the way on their own lines, (ii) the indented
lines prevented the multi-line string from getting mixed with
surrounding TOML parameters, and most importantly (iii) these
strings were processed by the Hugo Markdown parser, and so it wasn’t
sensitive to horizontal spaces.
And so the
tomelr-indent-multi-line-strings feature was born
(commit) which optionally made
tomelr export multi-line strings as
ox-hugo tests #
Once I had finalized the integration of
ox-hugo, I had
only about 30 tests change out of roughly 400 tests. These changes
were welcome as they fixed all the inconsistencies in the older TOML
generation code in
ox-hugo. If interested, you can see this commit
for the diff and details, but here’s the gist:
- Now nil value of a key in Lisp consistently implies that the key
should not be exported to TOML. So
'((foo . nil))will result in
foonot getting exported to TOML, whether that’s a top-level key or a key in a nested TOML map or array. If you need to set a key to a boolean false, use
"false"or any value from
- Earlier empty string value as in
'((foo . ""))behaved like the current nil implementation. That’s not the case any more. Now that empty string will export as
foo = ""in TOML.
- Now if a string has a quote character (
") in it, that value will auto-export as TOML multi-line string. I like the readability of this more than that of backslash-escaped double-quotes.
- Now the nested tables like
[menu."nested menu"]export with their parent table keys like
[menu]. As per the TOML spec, this is not required. But now that
tomelrhas added a generic support for any TOML table, this change happens as a result of consistency 💯.
In summary, the changes in
ox-hugo TOML front-matter exports were
mostly cosmetic, and if they were not cosmetic, they were consistency
What’s next? #
- The library is pretty much feature complete ✨ as
many of the examples from TOML v1.0.0 spec have been added to its
test suite, and .. it is supporting all the
The library though has one limitation that I’d like to resolve at some point — Right now, we require the Lisp data to first list all the scalar keys and then list the TOML tables and arrays of tables. But at the moment, I don’t know how to fix that, and also
ox-hugois not affected by that (because it already populates the front-matter alist in the correct order). So fixing this is not urgent, but of course, if someone can help me out with that, I’d welcome that! 🙏.
- Given that
tomelrallows robustly exporting any Lisp data expression to TOML, I do not see any value in continuing with YAML generation support using the old custom code.
📢 In near future, I plan to get rid of the
org-hugo-front-matter-formatcustomization variable from
ox-hugo— thus deprecating YAML export support This change should not functionally affect the YAML front-matter fans out there because the front-matter that
ox-hugois exporting is mainly for Hugo’s consumption. The only scenario where I see that this change can be breaking is if the user is using YAML format extra front-matter blocks. If so, unfortunately, they will need to convert those to TOML manually. and sticking with using just TOML for the front-matter.
Unblocking some future
ox-hugo improvements #
This decision will open up the doors to add more features to
:LOGBOOK:drawers to TOML front-matter (ox-hugo # 504)
Exporting Org Special Blocks to user-configurable front-matter (ox-hugo # 627)
Supporting more complex data in Lisp form using
:EXPORT_HUGO_CUSTOM_FRONT_MATTER:which could translate to nested TOML tables or arrays of TOML tables.
Finally, there won’t be a need to use the “Extra front-matter” workaround. For example, it would be possible to represent the data in that first example on that page as
:foo ((:bar 1 :zoo "abc") (:bar 2 :zoo "def"))in the