elisp on A Scripter's Notes

View GitHub Pull Requests in Magit

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Thu, 23 Jun 2022 17:51:00 -0400

How to view GitHub Pull Request branches locally in the cloned repo, and more importantly, how to do that automatically from within Emacs.

Table of Contents

Locally creating a branch for a PR
Getting references to all Pull Requests
Automatically adding PR refs
Summary
References

I have a few public projects in git repos, but I don’t get that much traffic in Pull Requests (PR) Gitlab calls these Merge Requests or MRs. . So when I need to add additional commits to a PR, I would just add the PR author’s remote to my local repo, check out their PR branch, add my own commits and then merge that to my project’s main branch.

As these occurrences were few and far apart, I didn’t have a need to view the PR branches directly from within Emacs/Magit. Though, I somehow knew that each GitHub Pull Request’s HEAD got assigned a git reference. But I didn’t need to use that knowledge until today 😃.

Today, when discussing PR # 20 on Prot’s Denote package’s GitHub mirror Prot uses SourceHut as the primary git forge for his Emacs packages. But I am glad that he doesn’t mind the activity in Issues and Pull Requests on the GitHub mirror. , he wrote this comment:

Now I just need to figure out how best to incorporate your changes into the org-id branch so I can add the final bits. I am not too familiar with the PR workflow …

.. and that inspired this post today.

Locally creating a branch for a PR #

From the GitHub docs, the git command to create a local branch for a PR is this:

git fetch origin pull/ID/head:BRANCHNAME

I have the denote package cloned from its GitHub mirror. So the origin remote’s url is https://github.com/protesilaos/denote.

Make sure that the remote name used in this command is pointing to a GitHub repo, and not a mirror forge like GitLab or SourceHut.

When I ran the below command, I got a new branch pr-20 pointing to the latest commit of that PR:

git fetch origin pull/20/head:pr-20

Awesome!

.. But that wasn’t good enough
.. Now I wanted more
.. I didn’t want to manually create a branch for each PR.

Getting references to all Pull Requests #

Now that I was on that quest of “I want more”, it didn’t take me long to re-discover this 7-year old nugget by Oleh Krehel. Here are the relevant bits from that post:

Open the local repo’s .git/config file.
Find the [remote "origin"] section

Modify it by adding this one line with pull refs. This is the same for all GitHub repositories.

[remote "origin"]
 url = https://github.com/USER/REPO.git
 fetch = +refs/heads/*:refs/remotes/origin/*
 fetch = +refs/pull/*/head:refs/pull/origin/*

With that edit in place, when I did l a (show the log for all git references), followed by f a (fetch all the remotes) in the Magit, I could see the references to the denote repo’s PRs!

Figure 1: Viewing PR references from denote package’s GitHub repo

.. But that still wasn’t good enough
.. I didn’t want to manually edit the .git/config in each repo.

Automatically adding PR refs #

Of course, I wasn’t the first one to think of this!

Another Emacs veteran Artur Malabarba had already had this covered also around 7 years back. Coincidentally, that post was written as a response to that same blog post by Oleh where he shared the above .git/config tip.

In that post, Artur shares an Emacs Lisp function that uses Magit functions like magit-get, magit-get-all and magit-git-string to auto-add the fetch = +refs/pull/*/head:refs/pull/origin/* line in the .git/config. This magic happens after checking that the origin remote points to a GitHub repo, and if that line doesn’t already exist.

Here, I am lightly modifying the function shared in that post so that the origin remote name is not hard-coded The reason is that sometimes, I name the original remote as upstream and my fork as fork, and I might have no remote named origin. . Credit for the main logic in this code still goes to Artur.

(defun modi/add-PR-fetch-ref (&optional remote-name)
 "If refs/pull is not defined on a GH repo, define it.

If REMOTE-NAME is not specified, it defaults to the `remote' set
for the \"main\" or \"master\" branch."
 (let* ((remote-name (or remote-name
 (magit-get "branch" "main" "remote")
 (magit-get "branch" "master" "remote")))
 (remote-url (magit-get "remote" remote-name "url"))
 (fetch-refs (and (stringp remote-url)
 (string-match "github" remote-url)
 (magit-get-all "remote" remote-name "fetch")))
 ;; https://oremacs.com/2015/03/11/git-tricks/
 (fetch-address (format "+refs/pull/*/head:refs/pull/%s/*" remote-name)))
 (when fetch-refs
 (unless (member fetch-address fetch-refs)
 (magit-git-string "config"
 "--add"
 (format "remote.%s.fetch" remote-name)
 fetch-address)))))
(add-hook 'magit-mode-hook #'modi/add-PR-fetch-ref)

Code Snippet 1: Function to auto-add GitHub PR references to the repo's .git/config

Summary #

With the above snippet added to your Emacs config and evaluated, each time you visit a repo cloned from GitHub in the Magit Status buffer (M-x magit-status), the PR refs will get auto-added to that repo’s .git/config if needed.

After that, you can easily view the commits from all the PRs by doing l a f a.

References #

Emacs Lisp: Advice Combinators

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Fri, 17 Jun 2022 21:30:00 -0400

My diagrammatic take on summarizing all the Emacs advice combinators.

Table of Contents

Overview on using advices
1 :before
2 :after
3 :override
4 :around
5 :before-while
6 :before-until
7 :after-while
8 :after-until
9 :filter-args
10 :filter-return
Summary
References

If you have read some of my earlier posts, you would know that I really enjoy using the Emacs Advice system 😃.

The “advice” feature lets you add to the existing definition of a function, by advising the function. This is a cleaner method than redefining the whole function, because it’s easier to debug and if you don’t need it, you can just remove the advice.

You can jump to the References section below if you need to look at the related sections in the Emacs Lisp Manual.

Overview on using advices #

I do not plan to write a tutorial on how to write advices in Emacs-Lisp, but here’s a 3-second primer:

To add an advice: (advice-add 'original-fn <combinator> #'advising-fn)
To remove an advice: (advice-remove 'original-fn #'advising-fn)

This article attempts to briefly describe different ways of advising a function, using 10 different combinators. If you have never used advices in your Emacs config, don’t worry. I am hopeful that the diagrams in this post and the examples linked for some of the combinators in the Summary section makes this concept a bit easier to assimilate.

Diagram Legend

Initial black circle: Original Fn input arguments
Yellow box: Original Fn
Gray box: Advising Fn
Black circle inside a white circle: Return values

1 `:before` #

Figure 1: :before advice

2 `:after` #

Figure 2: :after advice

3 `:override` #

Figure 3: :override advice

4 `:around` #

Figure 4: :around advice

5 `:before-while` #

Figure 5: :before-while advice

6 `:before-until` #

Figure 6: :before-until advice

7 `:after-while` #

Figure 7: :after-while advice

8 `:after-until` #

Figure 8: :after-until advice

9 `:filter-args` #

Figure 9: :filter-args advice

10 `:filter-return` #

Figure 10: :filter-return advice

Summary #

Here’s a concise summary of what each advice combinator does. For brevity, the advising function is called A and the original function is called O.

Once you click on any of the example posts, search for advice-add on that page to find the code example.

Table 1: Summary of what each advice combinator means

Combinator	Description	Example
`:before`	A is called before O. O args and return values are not modified.
`:after`	A is called after O. O args and return values are not modified.
`:override`	A is called in lieu of O. A gets the same args as O.	Zero HTML Validation Errors!
`:around`	A is called in lieu of O. A gets O fn + O args as args.	Using Emacs advice to silence messages from functions
`:before-while`	A is called first. If it returns non-nil, O is called.
`:before-until`	A is called first. If it returns nil, O is called.	Org: Show only Post subtree headings
`:after-while`	O is called first. If it returns non-nil, A is called.
`:after-until`	O is called first. If it returns nil, A is called.
`:filter-args`	A is called first. O is called next with return value from A as input.	Narrowing the Author column in Magit
`:filter-return`	O is called first. A is called next with return value from O as input.	Zero HTML Validation Errors!

If you have any feedback on how these diagrams can be made easier to understand, please let me know.

References #

Elisp Info: Advising Functions
- Elisp Info: Advice Combinators

Org: Show only Post subtree headings

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Thu, 16 Jun 2022 00:21:00 -0400

How to define a custom org-global-cycle-like command that collapses only the Org subtrees with specific properties.

Table of Contents

Org Global Cycle
Skeleton of only Post headings
The “Collapse All Posts” function
Binding with C-u C-c TAB
Result

I start this post by introducing what the Org mode global cycling command does, what kind of subtree folding I actually need, and then share the solution with code snippets.

Org Global Cycle #

Org mode has a built-in org-global-cycle command that you might be familiar with. It’s bound by default to the S-TAB key. Each time this command is called, the Org buffer will cycle through these states:

Overview: Show only the Level 1 headings and collapse everything underneath.
Contents: Show only the Org headings and collapse all the content.
Show All: Expand all the headings and show their contents too.

If a numeric prefix N is used with this command, it will show only the Org headings up to Level N. For example, C-2 S-TAB will show only the headings up to Level 2.

This is a really helpful command, but I needed something different ..

Skeleton of only Post headings #

I maintain most of this website’s content in a single Org file. I have dozens of blog posts organized in Org subtrees, which I further organize under “category” headings .. It kind of looks like the below mock-up: It’s amazing how many features PlantUML has. If you are interested in creating diagrams like these, check out the PlantUML Salt syntax.

Figure 1: Post Subtrees at arbitrary heading levels

As we can see,

All the post subtrees are not at Level 1 headings.
They are also not at a fixed Level N.
The heading level of the post depends on how many parent categories that post has (and that will also change over time).

I needed to basically show everything leading up to a post subtree heading, and then collapse all the content under that post; even the sub-headings.

The “Collapse All Posts” function #

The modi/org-hugo-collapse-all-posts function defined below meets the above requirement:

It first widens the whole buffer and expands all the headings.
Then it loops through all the headings and collapses all the post subtrees i.e. all the subtrees that have the EXPORT_FILE_NAME property set. This is where I use the org-map-entries magic.
Finally it looks for Org headings that begin with “Footnotes” or “COMMENT” and collapses them as well.

I am using the development version of Org mode (version 9.6, yet to be released as of <2022-06-15 Wed>) which has the new org-fold library. This library obsoletes the use of outline.el library and other code-folding related functions in Org mode. So cl-flet is used to create function symbol aliases that use the org-fold-* functions if available, otherwise they fall back to the legacy functions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


(defun modi/org-hugo-collapse-all-posts ()
 "Collapse all post subtrees in the current buffer.
Also collapse the Footnotes subtree and COMMENT subtrees if
present.

A post subtree is one that has the EXPORT_FILE_NAME property
set."
 (interactive)
 (cl-flet ((show-all (if (fboundp 'org-fold-show-all)
 #'org-fold-show-all
 #'org-show-all))
 (hide-subtree (if (fboundp 'org-fold-hide-subtree)
 #'org-fold-hide-subtree
 #'outline-hide-subtree)))
 (widen)
 (show-all '(headings))
 ;; Collapse all the post subtrees (ones with EXPORT_FILE_NAME
 ;; property set).
 (org-map-entries #'hide-subtree "EXPORT_FILE_NAME<>\"\"" 'file)
 ;; Also hide Footnotes and comments.
 (save-excursion
 (goto-char (point-min))
 (while (re-search-forward "^\\(\\* Footnotes\\|\\*+ COMMENT\\)"
 nil :noerror)
 (hide-subtree)))))

Code Snippet 1: Function that collapses all the post subtrees in the current buffer

Binding with `C-u C-c TAB` #

The function is ready, but let’s now add a bit of convenience to it.

If a point is under a subtree, C-c TAB will collapse that subtree while showing only Level 1 headings, and if a numeric prefix is used, it will show only those many levels of headings. I decided to bind the above function to C-u C-c TAB because,

The behavior of modi/org-hugo-collapse-all-posts falls in the same category as that of C-c TAB.
The C-u C-c .. binding rolls off the fingers pretty nicely 😃.

This binding is achieved using one of my favorite Emacs features .. the advice system. The :before-until Advice Combinator is used here, which means that if the advising function (below) returns a nil, the advised or the original function org-ctrl-c-tab is not called.

The advising function below detects if the C-u prefix argument is used. If it is, the modi/org-hugo-collapse-all-posts function is called, otherwise the original org-ctrl-c-tab function is called.

1
2
3
4
5
6
7
8
9


(defun modi/org-ctrl-c-tab-advice (&rest args)
 "Run `modi/org-hugo-collapse-all-posts' when
doing \\[universal-argument] \\[org-ctrl-c-tab]."
 (let ((do-not-run-orig-fn (equal '(4) current-prefix-arg)))
 (when do-not-run-orig-fn
 (modi/org-hugo-collapse-all-posts))
 do-not-run-orig-fn))

(advice-add 'org-ctrl-c-tab :before-until #'modi/org-ctrl-c-tab-advice)

Code Snippet 2: Bind C-u C-c TAB to call modi/org-hugo-collapse-all-posts

Result #

After evaluating the above two snippets, when I do C-u C-c TAB in my “blog posts” Org buffer, I see this:

Figure 2: My “blog posts” Org buffer showing only the Post subtree headings

It matches that earlier mockup — Mission accomplished! 💯

Looping through Org mode headings

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Wed, 18 May 2022 23:29:00 -0400

Using the org-map-entries API to loop through selected or all headings in an Org file.

Table of Contents

org-map-entries API
Example: Modifying a property in all headings
org-map-entries References

Below question on Mastodon by the user @postroutine inspired me to write this post:

I got a lot of Org-Mode headings and I want to modify their properties (add, remove, edit). Is there a function to do the same modifications on each heading?

I think that the best solution to that question is using the org-map-entries function.

But somehow when replying to that question then, that wasn’t what first came to my mind! .. when ironically that function is the main function that enables my preferred subtree-based flow in ox-hugo 😆. So I am writing this post to better ingrain the following concept in myself ..

If you need to loop through headings in an Org buffer, and especially if you need to modify that buffer in the process, use org-map-entries¹.

Next,

I will give a give introduction to the org-map-entries API.
Then provide a super-short solution to the above question.

`org-map-entries` API #

I will give only a broad level overview on how to use this function. I would encourage the reader to refer to the resources at the end of this post to learn more about it.

So let’s start by looking at this function’s signature:

(org-map-entries FUNC &optional MATCH SCOPE &rest SKIP)

Code Snippet 1: Signature of the org-map-entries function

The org-map-entries function iterates through all the headings meeting the MATCH criteria in the determined SCOPE, and then calls the specified function FUNC at each of those headings.

The FUNC function accepts no arguments and is called at the beginning of each Org heading.
The optional second argument MATCH is either nil, t or a search string.
- If MATCH is nil or t, all headings will be visited by the iteration and FUNC will be called on all of them.
- But if MATCH is a string, the headings will first be filtered based on that string and then the FUNC will be called on only those.
For explanations on the optional SCOPE and SKIP arguments, see Org Info: Using the Mapping API or C-h f org-map-entries from within Emacs.

Here’s a typical org-map-entries call that loops through all the headings in the visible buffer: (org-map-entries #'some-function) where all the optional argument values are nil. Next, we’ll see some examples of string-type MATCH arguments used for filtering the headings.

MATCH strings #

Below table shows few examples of match string patterns.

Table 1: String-type MATCH argument examples for org-map-entries

Search string	Description	Example
`"TAG"`	Tag name	`"foo"` matches all headings with that tag
`"{TAG REGEXP}"`	Regexp matching tags	`"{f.*}"` matches all headings whose tags match that regexp
`"TAG1+TAG2+.."`	Tag set intersection	`"foo+bar"` matches all headings with both of those tags
`"TAG1-TAG2+.."`	Tag set difference	`"foo-bar"` matches all headings with `foo` tag but without `bar` tag
`"TAG1`\|`TAG2`\|`.."`	Tag set union or boolean OR	`"foo`\|`bar"` matches all headings with either of those tags
`"TAG1&TAG2&.."`	Tag set intersection or boolean AND	`"foo&bar"` is same as `"foo+bar"`
`"PROP`=`\"STRVAL\""`	Specified property value matching a string	`"color`=`\"blue\""` matches all headings where `color` property is `blue`
`"PROP<>\"STRVAL\""`	Specified property value not matching a string	`"color<>\"blue\""` matches all headings where `color` property is not `blue`
`"PROP`=`{VAL REGEXP}"`	Specified property value matching a regexp	`"color={b.}"` matches all headings where `color` property value matches ‘`b.`’ regexp
`"PROP[OP]NUMVAL"`	Specified property value compared with a numeric value	`"some_num`>=`10"` matches all headings where `some_num` property is >=10
`"LEVEL[OP]VAL"`	Check value of headline’s special property LEVEL	`"level>2"` matches all headlines at levels greater than 2
`"TODO[OP]\"STRVAL\""`	Check value of headline’s TODO state	`"TODO`=`\"DONE\""` matches all headlines with TODO state set to ‘DONE’

Comparison Types #

If the comparison value is a plain number, a numerical comparison is done, and the allowed operators are ‘<’, ‘=’, ‘>’, ‘<=’, ‘>=’, and ‘<>’.
If the comparison value is enclosed in double quotes, a string comparison is done, and the same operators are allowed.
If the comparison value is enclosed in curly braces, a regexp match is performed. For this comparison, only ‘=’ (regexp matches) and ‘<>’ (regexp does not match) operators are allowed.
Comparison with dates and Group Tags is also possible. See Org Info: Matching tags and properties for more details.

Other notes #

The property names are case-insensitive. So these all work the same: "COLOR<>\"blue\"", "color<>\"blue\"", "Color<>\"blue\"".
The “tag” and “property” matches can be mixed up using the boolean ‘&’, ‘|’, ‘+’ and ‘-’ operators. So searching ‘+LEVEL=3+boss-TODO="DONE"’ lists all level three headlines that have the tag ‘boss’ and are not marked with the TODO keyword ‘DONE’.
‘&’ binds more strongly than ‘|’.
Grouping of match expressions using parentheses is not supported.

Example: Modifying a property in all headings #

Below is an example solution to the Mastoson question that I referenced in the beginning of this post.

(defun test/set-property-at-heading ()
 "Function to be called at the beginning of an Org heading."
 (let ((el (org-element-at-point)))
 (org-set-property "foo" (org-element-property :title el))))
(org-map-entries #'test/set-property-at-heading)

Code Snippet 2: Dummy example showing how to set a property for all Org headings using org-map-entries

It defines a function that parses the Org element at point using org-element-at-point, gets the title property of the element This function is designed to be called by org-map-entries and so the point at the time of calling this function will always be on a heading. , and sets that to the headline element’s foo property.
The org-map-entries call now simply calls this function on each heading in the visible scope of the Org buffer.

`org-map-entries` References #

Org mode has another popular mapping/looping API function org-element-map. I won’t go into much detail about that in this post — I’ll just mention that org-element-map is not the best choice if you need to modify the original Org buffer. It’s main use is to loop through a parsed AST of an Org buffer and optional modify those elements in memory. ↩︎

Presenting tomelr!

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Wed, 04 May 2022 01:03:00 -0400

In this post, I introduce a little library I created for ox-hugo to have a robust mechanism for generating TOML from any Lisp expression.

Table of Contents

My flow for creating this library
Adapting the library to fit ox-hugo
Changes in ox-hugo tests
What’s next?
- Unblocking some future ox-hugo improvements

In my previous post Defining tomelr, I started toying with the idea of creating a library that would help convert any Lisp expression to a TOML config, and I share my vision (specification) of what this library would look like.

I wasn’t even sure if I would be able to make the tomelr library feature-complete at least to the extent of what ox-hugo was already doing! But to my surprise, the library development snowballed to a completion much earlier than I thought, and additionally it helped fix some inconsistencies that the older TOML generation code had in ox-hugo.

In this post, I start by (i) giving a broad overview of how the development of tomelr happened, then (ii) briefly describe how it got integrated into ox-hugo, and finally (iii) how the use of this library will unblock the path to addition of some cool features to ox-hugo.

My flow for creating this library #

Write the spec for the library.
- List all the formats of Lisp data I would expect it to process.
- List the corresponding TOML data I would expect it to generate.
- Ensure that I am not inventing my own lisp syntax by confirming that the expected TOML output matches the JSON generated from that same lisp form (using the Emacs built-in json.el library).
That helped me write the tests first! – Test Driven Development (TDD).
I started with writing tests for TOML booleans and then implementing that (because that was the simplest and easiest). Of course, I used ert for this! ert helped me quickly create small modular tests Here’s the ert test for booleans as an example. and efficiently iterate through modifications in the library code until I got the tests to pass.
Once that got working, I set up a continuous integration system using GitHub Actions (GHA). I used GHA because I host my library on GitHub. Also I already have a tried and tested setup that I could get up and going in a matter of few seconds. In general, this concept would apply to any Continuous Integration system. The CI setup step should come early in the development of any project so that incremental feature additions don’t start breaking previously added features 😃.
The library development just snowballed after this point .. added support for integers, floats, regular strings, multi-line strings, arrays, TOML tables, arrays of TOML tables. By this time, the library was about 80% finished.
Then came the difficult part .. stabilizing the library to support all the varieties of Lisp data I can think of. I must have put in double the time spent so far to finish the remaining 20% of the planned features for this library 👉 tomelr test suite
Once I had the test suite complete and passing, it was time to do some code cleanup:
- Remove duplicate code and break them off into smaller helper functions.
- See if the function defined in this library is already defined somewhere else (in this case, I was able to use json-plist-p directly from json.el).
- Proof read the code.
- Proof read the docstrings and run M-x checkdoc to fix their formatting.
- Ensure that the code compiles without any warnings.
- Remove unnecessary customization options and case statements from the library (while continuously ensuring that the ert tests still pass).

Adapting the library to fit `ox-hugo` #

After polishing the library by its stand-alone testing, I decided to use it with ox-hugo and see how the test suite in that repo fared.

Of course I saw that a lot of tests failed now 😁.

The main issue was that tomelr was constructing multi-line strings such that the spaces translated exactly from Lisp data to TOML. So (tomelr-encode '((foo . "line1\nline2"))) would generate:

foo = """
line1
line2"""

Code Snippet 1: Multi-line string with same white-space as in original data.. but not that "pretty"

whereas ox-hugo expected the same TOML to look like:

foo = """
 line1
 line2
 """

Code Snippet 2: Pretty multi-line string, but with extra white-space

I had intentionally decided for ox-hugo to have this latter format for multi-line strings because (i) it made it more readable with the triple-quotes out of the way on their own lines, (ii) the indented lines prevented the multi-line string from getting mixed with surrounding TOML parameters, and most importantly (iii) these strings were processed by the Hugo Markdown parser, and so it wasn’t sensitive to horizontal spaces.

And so the tomelr-indent-multi-line-strings feature was born (commit) which optionally made tomelr export multi-line strings as expected by ox-hugo 😎.

Changes in `ox-hugo` tests #

Once I had finalized the integration of tomelr into ox-hugo, I had only about 30 tests change out of roughly 400 tests. These changes were welcome as they fixed all the inconsistencies in the older TOML generation code in ox-hugo. If interested, you can see this commit for the diff and details, but here’s the gist:

Now nil value of a key in Lisp consistently implies that the key should not be exported to TOML. So '((foo . nil)) will result in foo not getting exported to TOML, whether that’s a top-level key or a key in a nested TOML map or array. If you need to set a key to a boolean false, use "false" or any value from tomelr-false.
Earlier empty string value as in '((foo . "")) behaved like the current nil implementation. That’s not the case any more. Now that empty string will export as foo = "" in TOML.
Now if a string has a quote character (") in it, that value will auto-export as TOML multi-line string. I like the readability of this more than that of backslash-escaped double-quotes.
Now the nested tables like [menu."nested menu"] export with their parent table keys like [menu]. As per the TOML spec, this is not required. But now that tomelr has added a generic support for any TOML table, this change happens as a result of consistency 💯.

In summary, the changes in ox-hugo TOML front-matter exports were mostly cosmetic, and if they were not cosmetic, they were consistency fixes.

What’s next? #

tomelr: The library is pretty much feature complete ✨ as many of the examples from TOML v1.0.0 spec have been added to its test suite, and .. it is supporting all the ox-hugo use cases.
The library though has one limitation that I’d like to resolve at some point — Right now, we require the Lisp data to first list all the scalar keys and then list the TOML tables and arrays of tables. But at the moment, I don’t know how to fix that, and also ox-hugo is not affected by that (because it already populates the front-matter alist in the correct order). So fixing this is not urgent, but of course, if someone can help me out with that, I’d welcome that! 🙏.
ox-hugo: Given that tomelr allows robustly exporting any Lisp data expression to TOML, I do not see any value in continuing with YAML generation support using the old custom code.
📢 In near future, I plan to get rid of the org-hugo-front-matter-format customization variable from ox-hugo — thus deprecating YAML export support This change should not functionally affect the YAML front-matter fans out there because the front-matter that ox-hugo is exporting is mainly for Hugo’s consumption. The only scenario where I see that this change can be breaking is if the user is using YAML format extra front-matter blocks. If so, unfortunately, they will need to convert those to TOML manually. and sticking with using just TOML for the front-matter.

Unblocking some future `ox-hugo` improvements #

This decision will open up the doors to add more features to ox-hugo like:

Exporting Org :LOGBOOK: drawers to TOML front-matter (ox-hugo # 504)
Exporting Org Special Blocks to user-configurable front-matter (ox-hugo # 627)
Supporting more complex data in Lisp form using :EXPORT_HUGO_CUSTOM_FRONT_MATTER: which could translate to nested TOML tables or arrays of TOML tables.

Finally, there won’t be a need to use the “Extra front-matter” workaround. For example, it would be possible to represent the data in that first example on that page as :foo ((:bar 1 :zoo "abc") (:bar 2 :zoo "def")) in the :EXPORT_HUGO_CUSTOM_FRONT_MATTER: property.

Defining tomelr – A library for converting Lisp expressions to TOML

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Thu, 28 Apr 2022 00:08:00 -0400

Creating a specification for an Emacs-Lisp library to convert Lisp data expressions into easy-to-read TOML strings.

Table of Contents

Using json-encode as reference
Mapping scalar data to TOML
- About :false
Mapping lists to TOML
Mapping lists of lists to TOML
Mapping other object types
Closing

ox-hugo has some custom code that generates TOML I ❤️ TOML. As the makers of this config format put it.. “it’s a format for humans”! — No need to deal with indentations, weird syntax for multi-line strings, no prohibition on adding comments to the config, or dealing with careful placement of commas and braces. for Hugo front-matter, based on the Org keywords and other meta-data that the user sets in their Org file. But this TOML generation code is not generic enough for any TOML object type.

As I am writing up a definition on how to export :LOGBOOK: drawers, I felt it’s a good time to polish the whole Lisp data → TOML conversion code, and may be package that into a separate library.

It’s kind of an ambitious project — I am calling it tomelr ✨

It’s a big undertaking to create a generic library for this kind of data format conversion. But even before I start coding or think if I can complete this project, I need to spec I am not sure if it’s a widely used verb, but to spec means to write a specification for something. it. I need to understand early-on how the S-exp (Symbolic lisp expression) would needed to look for each kind of generated TOML object — scalars, lists, lists of lists, maps, lists of maps, etc.

Using `json-encode` as reference #

The aim of the tomelr library is to take some (lisp data) and convert that TOML. But I did not want to invent my own lisp data convention for this! So I decided to stick with the lisp expression conventions understood by the json-encode function from the Emacs core library json.el.

Credit for the json.el idea goes to this tweet by Piers Cawley:

I’d suggest that By “that”, he’s referring to adding support for exporting front-matter to JSON in ox-hugo. , since emacs has built in JSON support these days, you don’t really have to worry about the commas and braces, just build the s-exp you want and export, but it’s you that’s writing the code and I’m just delighted that it exists.

Thank you for your efforts.

I am not sold on adding support of yet another front-matter format to ox-hugo. I might not use json.el for that, but it definitely helped me a lot with coming up with this library’s spec 😆.

Mapping scalar data to TOML #

Figuring out the Lisp representation for scalar (plain key-value pairs) TOML objects was easy. json.el helped figure out how to deal with nil and false values.

Table 1: Mapping of scalar Lisp data to TOML

Lisp S-exp	TOML
`'((int_key . 123))`	`int_key = 123`
`'((float_key . 1.23))`	`float_key = 1.23`
`'((date_key . 2022-04-27))`	`date_key = 2022-04-27`
`'((bool_key . t))`	`bool_key = true`
`'((any_key . nil))`	(key removed in TOML)
`'((bool_key . :false))`	`bool_key = false`

About `:false` #

json.el defines a variable json-false that’s set to the value :json-false. This is because in JSON, the null value is different from the boolean false value.

nil in Lisp → null in JSON
:json-false in Lisp → false in JSON

Inspired by that decision of json.el, I am thinking of using :false as the special value that will set the equivalent TOML value to boolean false.

TOML does not define a null value, but if the Lisp value is nil, that key will simply not be translated to TOML.

Mapping lists to TOML #

Mapping lists was simple.. because in Lisp, a list value of course looks like '((foo . (1 2 3 4 5))) 😃.

Table 2: Mapping of list Lisp data to TOML

Lisp S-exp	TOML
`'((list_key1 . (1 2 3)))`	`list_key1 = [1, 2, 3]`
`'((list_key2 . ("a" "b" "c")))`	`list_key2 = ["a", "b", "c"]`

Mapping lists of lists to TOML #

Table 3: Mapping of list of list Lisp data to TOML

Lisp S-exp	TOML
`'((lol_key . [(1 2) (3 4)]))`	`lol_key = [ [1, 2], [3, 4] ]`

I was going to use '((lol_key . ((1 2) (3 4)))) as the reference Lisp expression for lol_key = [ [1, 2], [3, 4] ]. But I found out that json-encode throws an error if you pass it that expression! I don’t understand the reason for that error, and so I have asked for help on the help-gnu-emacs mailing list.

But while that question gets resolved, I wanted to move forward with the spec definition. After some trial-and-error and reverse-engineering json.el I knew how I wanted TOML to look. So I used an online JSON/TOML converter to convert that TOML snippet to JSON, and then used json-read to convert JSON to Lisp expression. , I learned that list of list data needs to be represented using a Vector type, and so '((lol_key . [(1 2) (3 4)])) would be the correct expression – Notice the use of square brackets instead of parentheses for the outer vector.

Mapping other object types #

Once I figured out how to map the above data types, mapping Lisp data to TOML Tables aka dictionaries and arrays of Tables was a breeze Of course, the breeze is referring to the ease of writing the spec for these 😆. Implementation-wise, the tables, arrays of tables, and the especially nested variants of those are going to be the most challenging. .

Closing #

It was fun coming up with an initial draft of the specification for this library. My next steps would be to gradually add TOML generator functions (as I find time) to this library, along with ERT tests! Eventually, I will remove the existing TOML generation code from ox-hugo and depend on this library.

Getting back to my plan for exporting :LOGBOOK: drawers in ox-hugo, based on this spec, I will need to construct this date in Emacs-Lisp:

(org_logbook . (((timestamp . 2022-04-08T14:53:00-04:00)
 (note . "This note addition prompt shows up on typing the `C-c C-z` binding.\nSee [org#Drawers](https://www.gnu.org/software/emacs/manual/html_mono/org.html#Drawers)."))
 ((timestamp . 2018-09-06T11:45:00-04:00)
 (note . "Another note **bold** _italics_."))
 ((timestamp . 2018-09-06T11:37:00-04:00)
 (note . "A note `mono`."))))

Code Snippet 1: Example data from a :LOGBOOK: drawer in Lisp format

.. will translate to this in TOML:

[[org_logbook]]
 timestamp = 2022-04-08T14:53:00-04:00
 note = """This note addition prompt shows up on typing the `C-c C-z` binding.
See [org#Drawers](https://www.gnu.org/software/emacs/manual/html_mono/org.html#Drawers)."""
[[org_logbook]]
 timestamp = 2018-09-06T11:45:00-04:00
 note = """Another note **bold** _italics_."""
[[org_logbook]]
 timestamp = 2018-09-06T11:37:00-04:00
 note = """A note `mono`."""

Code Snippet 2: Same example data from the :LOGBOOK: drawer translated to TOML format

Check out the below link where I have documented the equivalent expressions between Lisp, TOML and JSON for all the object types.

👉 tomelr Specification

Straight and Curved Quotes in Emacs Lisp

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Sun, 10 Apr 2022 10:05:00 -0400

A short guide for getting the single quotes rendered as expected (straight or curved) in Emacs Lisp function and variable documentation strings and message outputs.

Table of Contents

Emacs 25 – Curved quotes rendering feature
Preventing curved quotes in documentation strings
Preventing curved quotes in messages
Summary
References

This post is related to the curved quotes feature from Emacs 25 (September 2016), but I got inspired to write this as I was reading the NEWS file of the recently released Emacs 28.1 ✨ .. this specific part related to the shortdoc library. Turns out that I had never used shortdoc If you also haven’t used this library before, it’s basically a “cheat sheet generator” or an aggregator of related functions grouped together with code examples and their outputs – for example, functions related to string manipulation, buffer operations, etc. It is like https://tldr.sh/ but for Emacs Lisp functions. . So as I was reading through the help for various user-facing shortdoc functions, C-h f shortdoc-add-function led me to this:

Figure 1: Notice the curved closing single quotations in the Emacs Lisp example in that doc string

We don’t have curved quotes in Emacs Lisp syntax 🧐 .. So I look up the source of shortdoc-add-function:

1349
1350
1351
1352
1353
1354
1355
1356
1357
1358


(defun shortdoc-add-function (group section elem)
 "Add ELEM to shortdoc GROUP in SECTION.
If GROUP doesn't exist, it will be created.
If SECTION doesn't exist, it will be added.

Example:

 (shortdoc-add-function
 'file \"Predicates\"
 '(file-locked-p :no-eval (file-locked-p \"/tmp\")))"

Code Snippet 1: Unescaped single quotes in Emacs Lisp documentation strings

The documentation looks correct at first glance, but there’s a minor problem. This post talks about how to fix that documentation and then a bit more on how to deal with auto-conversion to curved quotes.

Emacs 25 – Curved quotes rendering feature #

Emacs 25 had landed with one of my favorite new features — the auto-rendering of grave accents (`) and straight quotes (') to curved (or curly) quotes (‘, ’). From its NEWS file:

New variable ’text-quoting-style’ to control how Emacs translates quotes. Set it to ‘curve’ for curved single quotes, to ‘straight’ for straight apostrophes, and to ‘grave’ for grave accent and apostrophe. The default value nil acts like ‘curve’ if curved single quotes are displayable, and like ‘grave’ otherwise. The new variable affects display of diagnostics and help, but not of info. As the variable is not intended for casual use, it is not a user option.

I like the rendering to curved quotes because it improves the readability of documentation strings in the ∗Help∗ buffers (C-h v, C-h f, etc.), and they look great in ∗Messages∗ buffer. It helps differentiate between typographical use of quotes like in ‘Hello’ versus the straight quotes used in Emacs Lisp syntax like (setq foo 'some-symbol) or the usage of grave accents like (setq foo `(,(+ 1 1))).

This feature renders grave accents (`) to ‘ and straight quotes (') to ’ by default. But there would be times when you would want to literally print grave accents or straight quotes in ∗Help∗ and ∗Messages∗ buffers.

Preventing curved quotes in documentation strings #

In documentation strings, some times you would want to show literal straight quotes or grave accents, for instance, when showing some example Emacs Lisp code snippets in there. Above image shows one of such examples.

More than the visual inaccuracy of seeing curved quoted where straight quotes should be, if someone copies that code to try it out, it will not work !

This should be fixed by escaping the ' characters.

Emacs Lisp uses a special escaping sequence in documentation strings: \\=.

Escape ` using \\=`.
Escape ' using \\='.

Below we see Code Snippet 1 after this fix:

1349
1350
1351
1352
1353
1354
1355
1356
1357
1358


(defun shortdoc-add-function (group section elem)
 "Add ELEM to shortdoc GROUP in SECTION.
If GROUP doesn't exist, it will be created.
If SECTION doesn't exist, it will be added.

Example:

 (shortdoc-add-function
 \\='file \"Predicates\"
 \\='(file-locked-p :no-eval (file-locked-p \"/tmp\")))"

Code Snippet 2: Correctly escaped single quotes in Emacs Lisp documentation strings

.. and now the straight quotes get rendered as straight quotes:

Figure 2: Correctly escaped single quotes in documentation strings

Preventing curved quotes in messages #

The same auto-rendering of curved quotes happens in the messages that get printed to the ∗Messages∗ buffer and Echo Area as well. So the below behavior where ‘Hi’ gets printed is expected and nice:

(message "`Hi'") ;=> "‘Hi’"

Code Snippet 3: message auto-converting `Hi' to ‘Hi’

But if someone does below, the rendered ’Hi’ looks pretty odd:

(message "'Hi'") ;=> "’Hi’"

Code Snippet 4: message auto-converting 'Hi' to ’Hi’ 😮

I had come across this surprise rendering in Emacs 25 and had raised this issue then. From Paul Eggert’s reply, I learned that to prevent this curved quote conversion, we should make the message function print the string processed by the format function and not the string directly. You could do this ..

(message "%s" (format "%s" "'Hi'")) ;=>"'Hi'"

Code Snippet 5: Passing format formatted string to message

But below will do the same thing and is much easier to type:

(message "%s" "'Hi'") ;=> "'Hi'"
(message "%s" "`Hi'") ;=> "`Hi'"

Code Snippet 6: Implicit passing of a string to format for formatting, before getting printed by message

Summary #

Table 1: Summary of techniques for enforcing straight quotes and grave accents in printed messages and documentation strings

Description	Solution	Outcome
Render `'` in docstring	doc string snippet: `"foo \\='bar"`	`foo 'bar`
Render ` in docstring	doc string snippet: "foo \\=`bar"	foo `bar
Print `'` as `'`	`(message "%s" "'foo")`	`'foo`
Print ` as `	(message "%s" "`foo")	`foo

By the way, I submitted a patch for fixing the escaping of straight quotes in shortdoc-add-function documentation string I planned to fix just this straight quote escaping issue, but then I also ended up slightly improving the documentation of the (FUNC :eval EVAL) and other forms used for adding a function’s documentation to shortdoc. in debbugs # 54782 and now it’s in the emacs master branch! 🎉

References #

Fixing curved quotes conversion in documentation strings

Elisp Info: Text Quoting Style

Fixing curved quotes conversion in printed messages

Elisp Info: Formatting Strings

Using Emacs advice to silence messages from functions

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Sat, 19 Mar 2022 02:25:00 -0400

Using the :around advice to prevent messages from being displayed in the echo area or the ∗Messages∗ buffer.

Table of Contents

Problem Statement
Course of action
1 Finding the culprit functions
- Using plain-old grep or ripgrep (rg)
- Using debug-on-message
2 Figure out how to silence those messages
3 Applying the ‘silence’ advice
- Limiting the scope of these advices
Conclusion

Today I was working on a minor cleanup of the ox-hugo package and as a part of that, I figured out a way to reduce the noisy messages printed in the ∗Messages∗ Random tidbit: You can quickly access the ∗Messages∗ buffer using the default binding C-h e. buffer. This post describes how I used the 📖 Emacs Advice feature to solve the noisy-messages-buffer problem.

What is this ‘advising’?: In Emacs, advising Advice with a ‘c’ is the noun form. The noun form is used everywhere in the Emacs advising API e.g. advice-add, advice-remove. Advise with an ’s’ is the verb form. More Reading. is a means to modify a function defined in any library — You can modify the arguments to the function, return values of that function, run other functions before and/or after that advised function, .. or may be just replace that function with something else entirely!

Problem Statement #

If I run C-c C-e H A This is the default binding of ox-hugo to export all the post subtrees in the current Org file. in this site’s Org file, I will see a wall of messages, which won’t be too useful to look at if I am exporting dozens of posts in one go.

Here’s a snippet from that which shows the text printed while two of the posts were being exporting:

[ox-hugo] 60/ Exporting ‘Nim: Deploying static binaries’ ..
org-babel-exp process nim at position 246918...
Evaluation of this nim code block is disabled.
org-babel-exp process shell at position 246985...
org-babel-exp process shell at position 247505...
org-babel-exp process shell at position 248054...
org-babel-exp process nim at position 250620...
Evaluation of this nim code block (code__musl_config_nims) is disabled.
org-babel-exp process yaml at position 253830...
Wrote /home/kmodi/hugo/scripter.co/content/posts/nim-deploying-static-binaries/index.md
[ox-hugo] 61/ Exporting ‘Binding Nim to C++ std::list’ ..
org-babel-exp process nim at position 261418...
Evaluation of this nim code block (code__nim_bindings_importcpp) is disabled.
org-babel-exp process nim at position 262802...
Evaluation of this nim code block (code__nim_std_list_bindings) is disabled.
org-babel-exp process cpp at position 266921...
org-babel-exp process nim at position 267957...
Evaluation of this nim code block (code__nim_bindings_emit) is disabled.
org-babel-exp process nim at position 268861...
Evaluation of this nim code block (code__nim_bindings_test_nim_code) is disabled.
org-babel-exp process text at position 269849...
org-babel-exp process nim at position 270412...
Evaluation of this nim code block (code__nim_bindings_toSeq) is disabled.
org-babel-exp process nim at position 271808...
Evaluation of this nim code block (code__nim_bindings_full_code) is disabled.
Wrote /home/kmodi/hugo/scripter.co/content/posts/binding-nim-to-c-plus-plus-std-list/index.md

Code Snippet 1: Snippet of the noise seen in the ∗Messages∗ buffer when exporting all post subtrees using ox-hugo

After applying the advices described in this post, that output reduced to this:

[ox-hugo] 60/ Exporting ‘Nim: Deploying static binaries’ ..
[ox-hugo] 61/ Exporting ‘Binding Nim to C++ std::list’ ..

Code Snippet 2: Snippet of the ∗Messages∗ buffer after applying the silencing advices

Course of action #

Here is how I tackled this problem:

Find the functions responsible for printing those messages.
Figure out how to silence those messages.
Advise those functions (temporarily, while the export is happening) to not print those messages.

1 Finding the culprit functions #

In Code Snippet 1, I have highlighted 2 lines. I first needed to figure out which functions printed those messages. I’ll show two methods of finding sources of any printed messages.

Using plain-old grep or ripgrep (rg) #

This method is pretty easy (but not robust) to use if the search string is unique enough. The first highlighted line above is a good candidate for this:

org-babel-exp process nim at position 246918…

The message gives me a clue that it’s originating somewhere in the Org Babel source code (one of the ob-*.el files). So I cd to the Org source directory and search for the “org-babel-exp process ..” string using rg ..

Figure 1: Finding the source of “org-babel-exp process ..” message using ripgrep (rg) in Org source

.. and sure enough, there is this one and only one hit .. in the org-babel-exp-src-block function in lisp/ob-exp.el.

But this method is not robust for some obvious reasons like,

the string being grepped might not be present in raw literal form like above,
may be it’s created through variable interpolation, or
may be that whole string is not present on the same line; may be it wrapped to the next line.

While this grep method helped find the source of “org-babel-exp process ..”, let’s use a better way of source tracing for the next message.

Using `debug-on-message` #

From C-h v debug-on-message, we see:

If non-nil, debug if a message matching this regexp is displayed.

This means that each time a message gets displayed that matches the regular expression set in this variable, we get a backtrace to that message. Who said we can get backtraces for errors only‽ 🤓

The second highlighted line from Code Snippet 1 is:

Wrote /home/kmodi/hugo/scripter.co/content/posts/nim-deploying-static-binaries/index.md

After evaluating (setq debug-on-message "Wrote"), when I exported this post, I saw a backtrace that looked like this.

Debugger entered--Lisp error: "Wrote /home/kmodi/hugo/scripter.co/content/posts/u..."
 write-region(nil nil "/home/kmodi/hugo/scripter.co/ ..
 basic-save-buffer-2()
 basic-save-buffer-1()
 ..
 basic-save-buffer(nil)
 save-buffer()
 write-file("/home/kmodi/hugo/scripter.co/content/posts/using-e...")
 ..
 org-export-to-file(hugo "/home/kmodi/hugo/scripter.co/content/posts/using-e..." ..
 org-hugo-export-to-md(nil :subtreep nil)
 ..

Code Snippet 3: Backtrace created on setting debug-on-message to "Wrote"

The top line of that backtrace shows the source of that message: write-region.

Remember to reset debug-on-message back to nil by evaluating (setq debug-on-message nil) once you are done with this debug!

2 Figure out how to silence those messages #

Now that I knew that I needed to alter the org-babel-exp-src-block and write-region functions There was a third function too that I found using this second method: table-generate-source. This was printing “Generating source…” when parsing each table.el message. , I wanted to quiet these messages in both the echo area and the ∗Messages∗ buffer, but only when exporting all the post subtrees from a file.

Emacs provides us ways to separately control the displaying of messages in those regions:

Prevent messages in the echo area: From 📖 Elisp – Displaying Messages You can visit the same manual page from within Emacs by C-h i g (elisp)Displaying Messages. , if inhibit-message is set to t , message outputs are not shown in the echo area (though they would still be logged in the ∗Messages∗ buffer.
Prevent messages in the ∗Messages∗ buffer: From 📖 Elisp – Logging Messages C-h i g (elisp)Logging Messages , if message-log-max is set to nil , no messages will be printed in the ∗Messages∗ buffer.

3 Applying the ‘silence’ advice #

With that information, I created this advising function which calls the function name it receives in its first argument.

1
2
3
4
5


(defun org-hugo--advice-silence-messages (orig-fun &rest args)
 "Advice function that silences all messages in ORIG-FUN."
 (let ((inhibit-message t) ;Don't show the messages in Echo area
 (message-log-max nil)) ;Don't show the messages in the *Messages* buffer
 (apply orig-fun args)))

Code Snippet 4: Advising function to silence the messages in the advised orig-fun

Note that the above function does not set the inhibit-message and message-log-max variables globally (e.g. using setq). We do not want to permanently stop the printing of messages for all the functions! That’s why they are set in the let scope or the local scope, and the orig-fun is called within that scope.

This snippet, though, only defines the advising function. The advices are actually applied using the advice-add The counterpart of advice-add is advice-remove, to remove the advising functions from the original functions. function.

Emacs provides a variety of Advice Combinators. These combinators are used to specify how an advice should be applied to the original function. I have found the :around combinator to usually fit my needs because the advices I add typically look like:

Do something before.
Set some variables in a let scope.
Call the original function with its arguments inside that scope.
Do something afterwards.

As I wanted to apply the same advice to multiple functions (org-babel-exp-src-block and write-region), I used dolist to loop through them and advise them to 🤫 :

1
2


(dolist (fn '(org-babel-exp-src-block write-region)
 (advice-add fn :around #'org-hugo--advice-silence-messages))

Code Snippet 5: Enabling the advices

Limiting the scope of these advices #

You might wonder — If ox-hugo is applying these advices, wouldn’t it affect the global behavior of write-region, etc. outside of ox-hugo exports as well?

The answer is “no”.

In ox-hugo, the advice-add calls are made just before the “export all post subtrees” operation begins. Once the export command finishes, advice-remove calls remove those advices from those functions. So those advices are effective only for the duration of the export.

Details of the implementation can be seen in this ox-hugo commit.

Conclusion #

The main effort here was to (i) identify the problem, (ii) find out which functions were causing the problem, and (iii) what knobs or variables are available to us to help fix that.

Once that is done, you write an advising function and advice-add that function to the original function using a combinator of your choice If you are not sure which combinator to use, pick :around 😃. .

In this post, I am using Emacs advices to make ox-hugo exports less noisy. But you can add similar advices to your Emacs config You can find the ways I am using advices in my Emacs config here. to tweak and fine-tune any function that’s getting on your nerves 😄.

Quick Intro to Emacs Lisp Regression Testing

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Sun, 06 Mar 2022 14:40:00 -0500

A quick introduction to using ERT (Emacs Lisp Regression Testing) for your next elisp library.

Table of Contents

Test directory
Test Files
- Feature-specific t***.el test files
- Wrapper test file all-tests.el
Makefile
Summary
References

ERT is a testing library that comes with Emacs i.e. you do not need to install from GNU Elpa or Melpa. The aim of this post is to serve as a quick guide on how to start using ERT with your Emacs Lisp library Here, a “library” could even be a set of custom Emacs Lisp code that you wrote for your Emacs config. .

The ERT manual You can quickly visit the ERT manual from within Emacs using the wonderful Info system by typing C-h i m ert, you can visit Emacs Manual – ERT in a web browser. is really well-written and here’s how it introduces this testing library:

ERT is a tool for automated testing in Emacs Lisp. Its main features are facilities for defining tests, running them and reporting the results, and for debugging test failures interactively.

..it works well both for test-driven development and for traditional software development methods.

The part about test-driven development is so true! At the moment, I am developing an Emacs library called baser that does signed number conversions among base 10, base 2 and base 16 formats. Before adding new features, I first write the ert tests, and then I develop and refine the functions until they start passing.

In this post, I will use that baser library as an example of how to set up ERT – which I believe would be applicable to other Emacs Lisp projects too.

Test directory #

Below shows how the test/ directory is created with respect to the library being tested.

baser/ (repo root)
├── baser.el
├── ..
├── test/
│ ├── all-tests.el
│ ├── ..
│ └── tdec-hex.el
└── Makefile

Code Snippet 1: Test directory structure from baser

Here,

baser.el is the Emacs Lisp library being tested.
test/ is the directory containing all the tests.
Makefile defines a test target so that running tests is as easy as make test.

Test Files #

In the test/ directory, I have a main all-tests.el file that requires all other test files named in t***.el style.

Feature-specific `t***.el` test files #

I like to follow the convention of starting all test file names with t.

I break up the test files such that each file tests only a particular feature. In baser, I have tdec-hex.el to test the conversions between decimal and hexidecimal formats. Similarly I have tdec-bin.el and thex-bin.el.

General structure of each feature test #

The structure of each tfeature.el file looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


;; tfeature.el -*- lexical-binding: t; -*-

;; Require the emacs-lisp library being tested.
;; Example: (require 'baser)

;; Test things that *should* happen.
(ert-deftest addition-test ()
 (should (= (+ 1 2) 3)))

;; Test things that your library *should trigger errors* for.
(ert-deftest div-by-0-test ()
 (should-error (/ 0 0)
 :type 'arith-error))


(provide 'tfeature)

Code Snippet 2: Structure of each tfeature.el test file

Lines 7 and 11 define the ert tests using ert-deftest.
The first ert_deftest uses a should macro that checks if its body evaluates to non-nil. It is similar to constructs like assert (something == 1); or doAssert something == true in other programming languages.
The second ert_deftest uses kind the should-error macro that asserts if the evaluation of its body resulted in an error. Here, the test fails if the body does not throw an error. The optional :type <error type> argument makes the test stricter; in addition to checking if evaluating that body throws an error, it also checks if the thrown error is of that specific type.

ert does not need to be required manually because ert-deftest will autoload it.

Wrapper test file `all-tests.el` #

A wrapper test file all-tests.el is created for convenience so that loading just this one file loads all the tests. Having such a wrapper file is helpful as you will see in the Makefile section next.

This file simply requires all the feature test files like the ones explained above.

;; all-tests.el

;; If the directory happens to have both compiled and uncompiled
;; version, prefer to use the newer (typically the uncompiled) version.
(setq load-prefer-newer t)

(require 'tdec-hex) ;Require the dec<->hex conversion feature test
;; more feature tests ..

Code Snippet 3: Wrapper test file all-tests.el to require all other tests

Makefile #

Finally, we create a Makefile so that we can run make test to run the ERT tests.

Run make test to run all tests.
Run make test MATCH=foo to run only the tests whose names match “foo”. The test names are the ones defined by the ert-deftest macro.

EMACS ?= emacs

TEST_DIR=$(shell pwd)/test

# Run all tests by default.
MATCH ?=

.PHONY: test

test:
 $(EMACS) --batch -L . -L $(TEST_DIR) -l all-tests.el -eval '(ert-run-tests-batch-and-exit "$(MATCH)")'

Code Snippet 4: make test recipe for running ERT tests

See ERT – Running Tests in Batch Mode for reference.

Summary #

You can quickly create a nice test setup for your Emacs Lisp library or Emacs config by adding a test file with tests defined using erf-deftest with should and should-error macros, and then running make test with the help of a Makefile like the one shown above.

While this post might help you get started with using ERT quickly, do go through its manual — it’s short and sweet, and well-written, and covers a lot more than this post.

References #

Splitting an Org block into two

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Wed, 22 Aug 2018 17:47:00 -0400

I ventured out to start writing about a 100+ line Emacs Lisp snippet in my config, and then I thought — Wouldn’t it be nice if I can quickly split out that huge snippet into smaller Org Src blocks?

And so this blog post happened.

Table of Contents

Problem
Action Plan
Am I in an Org block?
If so, split the block
Now make M-return do that
Full code

Problem #

If I have a huge Org Src block, I’d like to split it into multiple Org Src blocks so that I can write my explanations in-between.

So I’d like to quickly split up:

#+begin_src emacs-lisp
(message "one")
(message "two")
#+end_src

into:

#+begin_src emacs-lisp
(message "one")
#+end_src

#+begin_src emacs-lisp
(message "two")
#+end_src

.. like this.

☝ Click for animation.

Action Plan #

Write a function to return non-nil if point is in any Org block – Not just “src”, “example”, “export” or any of the inbuilt Org blocks.. but also any Org Special block like #+begin_foo .. #+end_foo.
Write a function that does this imagined block splitting.
Overload the M-return binding so that this block splitting function gets called only when the point is inside an Org block (detected using that first function).

<2018-08-26 Sun> Thanks to the comment by reader Mankoff, I learnt about the org-babel-demarcate-block function (bound by default to C-c C-v d and C-c C-v C-d).

This function varies from the solution in this post in at least two ways:

It works only for Org Src blocks.
It splits the block exactly at where the point is, whereas I would like to always split only at EOL or BOL.

But I can see that org-babel-demarcate-block can cover most of the block splitting use cases.

Am I in an Org block? #

Before venturing into writing this function, I looked at these existing ones, but none did what I exactly wanted:

org-in-src-block-p: Returns non-nil only if the point is in a #+begin_src .. #+end_src block; not when point is in any other Org block.
org-in-block-p: Returns non-nil only if the point is in one of the pre-defined block names passed as a list ('("src" "example" "quote" ..)). So this again won’t work as I cannot pre-define all Org Special blocks.

So I define the below modi/org-in-any-block-p function that returns non-nil if the point is in-between any #+begin_FOOBAR .. #+end_FOOBAR. Thankfully, I was able to reuse a lot of logic from the org-between-regexps-p function (org-in-block-p uses that function internally).

(defun modi/org-in-any-block-p ()
 "Return non-nil if the point is in any Org block.

The Org block can be *any*: src, example, verse, etc., even any
Org Special block.

This function is heavily adapted from `org-between-regexps-p'."
 (save-match-data
 (let ((pos (point))
 (case-fold-search t)
 (block-begin-re "^[[:blank:]]*#\\+begin_\\(?1:.+?\\)\\(?: .*\\)*$")
 (limit-up (save-excursion (outline-previous-heading)))
 (limit-down (save-excursion (outline-next-heading)))
 beg end)
 (save-excursion
 ;; Point is on a block when on BLOCK-BEGIN-RE or if
 ;; BLOCK-BEGIN-RE can be found before it...
 (and (or (org-in-regexp block-begin-re)
 (re-search-backward block-begin-re limit-up :noerror))
 (setq beg (match-beginning 0))
 ;; ... and BLOCK-END-RE after it...
 (let ((block-end-re (concat "^[[:blank:]]*#\\+end_"
 (match-string-no-properties 1)
 "\\( .*\\)*$")))
 (goto-char (match-end 0))
 (re-search-forward block-end-re limit-down :noerror))
 (> (setq end (match-end 0)) pos)
 ;; ... without another BLOCK-BEGIN-RE in-between.
 (goto-char (match-beginning 0))
 (not (re-search-backward block-begin-re (1+ beg) :noerror))
 ;; Return value.
 (cons beg end))))))

Code Snippet 1: Function to check if point is in any Org block

(case-fold-search t) ensures that either #+BEGIN_ .. or #+begin_ .. match.
The regular expression in block-begin-re matches with "#+begin_src foo" or " #+begin_src foo" or "#+BEGIN_EXAMPLE" or "#+begin_FOOBAR" or ..
The limit-up and limit-down are set to the buffer locations of the previous and next Org headings. The following regexp searches are limited to happen in those bounds for better performance.
The block-end-re is dynamically constructed based on the string matched using block-begin-re. This is so that if "#+begin_quote" is found initially, it matches the block ending with specifically "#+end_quote" and not something like "#+end_src".
nil is returned if the point is not between #+begin_FOOBAR .. #+end_FOOBAR.

Caveat

I haven’t gone extra lengths to support nested block cases, specifically where the point is outside the inner-most block, but still inside the outer block:

#+begin_src org
▮
#+begin_src emacs-lisp
(message "hello!")
#+end_src
#+end_src

If so, split the block #

With the “point in an Org block” detection working, I now needed the split to happen with these rules:

If the point is anywhere on the line, but not at the beginning of the line (BOL),
- Go to the end of the line, and then split the block.
  
  So if the point¹ is after the first message identifier, or at the end of that first message line:
```
#+begin_src emacs-lisp
(message "one")▮
(message "two")
#+end_src
```
  Split the block at the point after (message "one") and move the point to between the split blocks:
```
#+begin_src emacs-lisp
(message "one")
#+end_src
▮
#+begin_src emacs-lisp
(message "two")
#+end_src
```
Otherwise (if point is at BOL),
- Split the block exactly at that point.
  
  So if the point is at the beginning of the second message line:
```
#+begin_src emacs-lisp
(message "one")
▮(message "two")
#+end_src
```
  Split the block at the point before (message "two") and move the point to between the split blocks:
```
#+begin_src emacs-lisp
(message "one")
#+end_src
▮
#+begin_src emacs-lisp
(message "two")
#+end_src
```

So here’s the code that follows that spec:

(defun modi/org-split-block ()
 "Sensibly split the current Org block at point."
 (interactive)
 (if (modi/org-in-any-block-p)
 (save-match-data
 (save-restriction
 (widen)
 (let ((case-fold-search t)
 (at-bol (bolp))
 block-start
 block-end)
 (save-excursion
 (re-search-backward "^\\(?1:[[:blank:]]*#\\+begin_.+?\\)\\(?: .*\\)*$" nil nil 1)
 (setq block-start (match-string-no-properties 0))
 (setq block-end (replace-regexp-in-string
 "begin_" "end_" ;Replaces "begin_" with "end_", "BEGIN_" with "END_"
 (match-string-no-properties 1))))
 ;; Go to the end of current line, if not at the BOL
 (unless at-bol
 (end-of-line 1))
 (insert (concat (if at-bol "" "\n")
 block-end
 "\n\n"
 block-start
 (if at-bol "\n" "")))
 ;; Go to the line before the inserted "#+begin_ .." line
 (beginning-of-line (if at-bol -1 0)))))
 (message "Point is not in an Org block")))

Code Snippet 2: Function to split the current Org block in sensible fashion

The regexp for extracting block-start is the same as block-begin-re in Code Snippet 1, but with different sub-grouping.
The block-end string is derived from sub-group 1 of block-start string – just replacing “begin_” with “end_”.
And then based on if the point was initially at BOL (at-bol), the insertion of newlines and movement of point is done accordingly.

Now make `M-return` do that #

With these two functions evaluated, M-x modi/org-split-block will work right away.

But where’s the fun in that‽

I needed to have the Org block splitting happen with an intuitive binding — like M-return.

By default, M-return is used to either create new headings, or do other things like insert an item, wrap a region in table, etc. based on the context. See the doc-string of org-meta-return (function bound to this key by default) for more info.
But it doesn’t have a context for “point in an Org block”. So it tries to create a heading when inside a block too, which doesn’t make much sense.
So fix that by adding that context.

So I advise org-meta-return to call modi/org-split-block when the point is inside an Org block.

The advising function modi/org-meta-return is the same as the advised function org-meta-return (as of <2018-08-26 Sun>), except that a new context (modi/org-in-any-block-p) is added.

You can tweak the precedence of this new context by moving the ((modi/org-in-any-block-p) #'modi/org-split-block) form in that cond form.

(defun modi/org-meta-return (&optional arg)
 "Insert a new heading or wrap a region in a table.

Calls `org-insert-heading', `org-insert-item',
`org-table-wrap-region', or `modi/org-split-block' depending on
context. When called with an argument, unconditionally call
`org-insert-heading'."
 (interactive "P")
 (org-check-before-invisible-edit 'insert)
 (or (run-hook-with-args-until-success 'org-metareturn-hook)
 (call-interactively (cond (arg #'org-insert-heading)
 ((org-at-table-p) #'org-table-wrap-region)
 ((org-in-item-p) #'org-insert-item)
 ((modi/org-in-any-block-p) #'modi/org-split-block)
 (t #'org-insert-heading)))))
(advice-add 'org-meta-return :override #'modi/org-meta-return)

Code Snippet 3: Advising org-meta-return to add context of point being inside any Org block

Now with the point in any Org block, M-return away!

Full code #

Look for the source of modi/org-split-block (and dependent functions) added to setup-org.el in my Emacs config.

The point is denoted by the BLACK VERTICAL RECTANGLE unicode char (▮). ↩︎

Optimize your FontAwesome

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Wed, 04 Apr 2018 15:20:00 -0400

How to trim the FontAwesome JS for your website, using Emacs Lisp.

Table of Contents

FontAwesome 5 Setup
Problem
Solution
Code
Result

This post briefly goes through the FontAwesome 5 setup, the problem statement and a solution for that.

I signed up for the FontAwesome 5 Pro Kickstarter, and I have to say that this is one of my few Kickstarter fundings that delivered really well!

Thank you FontAwesome!

FontAwesome 5 Setup #

FontAwesome 5 supports an awesome new way of loading icons, using SVG with JS. The benefit is that you simply load one fontawesome-all.js file and everything just works!.

<head>
 <!--load everything-->
 <script defer src="/static/fontawesome/fontawesome-all.js"></script>
</head>
<body>
 <!--icon-->
 <i class="fas fa-user"></i>
</body>

Code Snippet 1: Fontawesome 5 Setup

Problem #

With FontAwesome 5 Pro, I get about 2700 icons packed in that fontawesome-all.js. While that is great, there are 2 issues:

I don’t need all of those 2700 icons on my site. That file is roughly 1.7 MB (minified or not) which I need not unnecessarily load for all my visitors.
Anyone can easily pirate the whole JS file and use it for free!

Solution #

The solution is simple — I just comment out all the lines with SVG code for the icons that I don’t use.

But searching for the icons I need in all the icon packs (Solid, Regular, Light, Brand) in that single 5k+ line JS file is not fun, especially when I want to add/remove icons occasionally.

And so fontawesome-choose was born!

I list the icons I need in fontawesome-choose-icons.
Run M-x fontawesome-choose in the fontawesome-all.js file buffer.

The code is in the next section, but you can also find the latest version in my repo.

Note that while I have used this on Fontawesome 5 Pro, it should work just as well on Fontawesome 5 Free too.

Code #

;; Helper function `fontawesome-choose' used to uncomment only the
;; icons the user cares about in `fontawesome-all.js'.

(defconst fontawesome-choose-icons '("list-alt" ;categories
 "tags"
 "rss"
 "link"
 "heart" ;like
 "reply"
 "retweet"
 "github" ;"github-alt" "github-square"
 "twitter" ;"twitter-square"
 "gitlab")
 "List of icons to choose from fontawesome-all.js.
Used in `fontawesome-choose'.")

(defun fontawesome-choose ()
 "Comment out all icons in fontawesome-all.js except the selected few.

Minifying the resultant .js will then remove the commented icons,
thus drastically reducing the minified JS size.

Set the `fontawesome-choose-icons' variable to the list of icons
that you want to keep uncommented."
 (interactive)
 (let ((case-fold-search nil)
 (count 0))
 (save-excursion
 (goto-char (point-min))
 (while (re-search-forward "^var icons" nil :noerror)
 (let ((begin (progn
 (forward-line 1)
 (point)))
 end)
 (re-search-forward "^\\};")
 (forward-line 0)
 (backward-char 1)
 (setq end (point))
 ;; First comment all the lines
 (save-excursion
 (narrow-to-region begin end)
 (goto-char (point-min))
 (while (re-search-forward "^\\s-*\\(\"\\)" nil :noerror)
 (replace-match "// \\1" nil nil nil 1))
 (widen))
 ;; Now uncomment only the selected icons
 (save-excursion
 (narrow-to-region begin end)
 (goto-char (point-min))
 (let* ((icon-regexp (regexp-opt fontawesome-choose-icons 'symbols))
 (regexp (format "^\\s-*\\(//\\s-*\\)\"%s\":" icon-regexp)))
 (while (re-search-forward regexp nil :noerror)
 (replace-match "" nil nil nil 1)
 (setq count (1+ count))))
 (widen))))
 (message "fontawesome-choose: Uncommented %d icons matching %S"
 count fontawesome-choose-icons))))

Code Snippet 2: fontawesome-choose — Comment out all the FontAwesome icons you don't need

Result #

This solution solves the first problem wonderfully — After minification, fontawesome-all.js is just 71 KB (down from 1.7 MB).

That’s a 25x factor reduction! The minification step is important because that removes all the commented lines from the JS. I do the minification using tdewolff/minify.

The second problem, that of piracy, is difficult to solve completely using this. But whoever pirates this reduced FontAwesome from my site won’t get the real deal 😎.

Emacs-Lisp saves the day once again!

Your car will be ready in 8000 seconds

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Wed, 07 Feb 2018 18:10:00 -0500

Table of Contents

Code
Tests
- Test Output
Source
Closing
Appendix
- Test output using format-seconds
- Code (Revision 1)

Huh? 😕

Well, a mechanic usually wouldn’t give you a time estimate in seconds, but a tool I am using prints something like this at the end:

The simulation took 54227.9 seconds in CPU time.

That triggered me to write a “little” script to convert seconds to human time i.e. time in days, hours, minutes and seconds.

Thanks to /u/xiongtx from Reddit, I learned about the built-in function format-seconds that does what I wanted to do – but not exactly in a way I wanted to see. Though, format-seconds gave me an idea for a big optimization (code commit diff).

Below Code section is updated to reflect that. If you like, you can review older version of the same section at the end of this post. Also, at the end, you will find a comparison between the outputs from format-seconds and modi/seconds-to-human-time.

Code #

Here’s the updated code, and notes about that follow after that:

(defun modi/seconds-to-human-time (&optional seconds)
 "Convert SECONDS to \"DDd HHh MMm SSs\" string.

SECONDS is a non-negative integer or fractional number.

SECONDS can also be a list of such numbers, which is the case
when this function is called recursively.

When called interactively, if a region is selected SECONDS is
extracted from that, else the user is prompted to enter those."
 (interactive)
 (let ((inter (called-interactively-p 'interactive)))
 (when inter
 (let ((seconds-str (if (use-region-p)
 (buffer-substring-no-properties (region-beginning) (region-end))
 (read-string "Enter seconds: "))))
 (setq seconds (string-to-number seconds-str)))) ;"1" -> 1, "1.2" -> 1.2, "" -> 0
 (let* ((MINUTE 60)
 (HOUR (* 60 MINUTE))
 (DAY (* 24 HOUR))
 (sec (cond
 ((listp seconds) ;This is entered only by recursive calls
 (car (last seconds)))
 ((and (numberp seconds) ;This is entered only in the first entry
 (>= seconds 0))
 seconds)
 (t
 (user-error "Invalid argument %S" seconds))))
 (gen-time-string
 (lambda (time inter)
 "Return string representation of TIME.
TIME is of the type (DD HH MM SS), where each of those elements
are numbers. If INTER is non-nil, echo the time string in a
well-formatted manner instead of returning it."
 (let ((filler " ")
 (str ""))
 (dolist (unit '("d" "h" "m" "s"))
 (let* ((val (car (rassoc unit time)))
 (val-str (cond
 ((and (string= unit "s") ;0 seconds
 (= val 0)
 (string-match-p "\\`\\s-*\\'" str))
 " 0s")
 ((and (string= unit "s")
 (> val 0))
 (if (integerp val)
 (format "%2d%s" val unit)
 (format "%5.2f%s" val unit)))
 ((and val (> val 0))
 (format "%2d%s " val unit))
 (t
 filler))))
 (setq str (concat str val-str))))
 ;; (message "debug: %S" time)
 (if inter
 (message "%0.2f seconds → %s"
 seconds
 (string-trim (replace-regexp-in-string " +" " " str)))
 (string-trim-right str)))))
 (time (cond
 ((>= sec DAY) ;> day
 (let* ((days (/ (floor sec) DAY))
 (rem (- sec (* days DAY))))
 ;; Note that (list rem) instead of just `rem' is
 ;; being passed to the recursive call to
 ;; `modi/seconds-to-human-time'. This helps us
 ;; distinguish between direct and re-entrant
 ;; calls to this function.
 (append (list (cons days "d")) (modi/seconds-to-human-time (list rem)))))
 ((>= sec HOUR) ;> hour AND < day
 (let* ((hours (/ (floor sec) HOUR))
 (rem (- sec (* hours HOUR))))
 (append (list (cons hours "h")) (modi/seconds-to-human-time (list rem)))))
 ((>= sec MINUTE) ;> minute AND < hour
 (let* ((mins (/ (floor sec) MINUTE))
 (rem (- sec (* mins MINUTE))))
 (append (list (cons mins "m")) (modi/seconds-to-human-time (list rem)))))
 (t ;< minute
 (list (cons sec "s"))))))
 ;; If `seconds' is a number and not a list, this is *not* a
 ;; recursive call. Return the time as a string only then. For
 ;; re-entrant executions, return the `time' list instead.
 (if (numberp seconds)
 (funcall gen-time-string time inter)
 time))))

Code Snippet 1: Seconds to Human Time

Most of this snippet is just the day/hour/minute/second math. Apart from that, here are some points that I found of interest:

I did not always want to prompt the user to enter the input argument. If a region was selected, the function assumes that the user selected a number, and skips the prompt step. So I used a plain (interactive) form instead of using (interactive "sPrompt: ") or (interactive "r"). See (eintr) Interactive Options and (elisp) Interactive Codes to learn about interactive and its codes.
Instead of in-lining a modular chunk of logic, like the one where I convert a list like (1 2 3 4) into "1d 2h 3m 4s", I assigned it to a let-bound symbol gen-time-string. That allowed the logic to be more discernible when used in:
```
(if (numberp seconds)
 (funcall gen-time-string time inter)
 time)
```
- Also interesting is the fact that these let-bound lambdas can have their own doc-strings too.
I make use of recursion in this function! But I needed this function to return a string (using that gen-time-string function) only when all the nested calls to itself were returned. So to distinguish between a direct call to the function, and re-entrant calls, when doing the latter, I make the input number a list of that number.
- So while the function might take an input number like 7 for a direct call, that same number, when needed to call to a recursive call, would get passed as (list 7) or '(7).
- If you glance back as that little snippet above, I return the time as a string only if the input seconds is a number — and not a list i.e. only when I am in the “direct call instance”.
The internal variable time is now an alist and can have up to 4 cons elements. Each cons is of the type (TIMEVALUE . TIMEUNIT). So time now looks like ((DAYS . "d") (HOURS . "h") (MINUTES . "m") (SECONDS . "s")).

If the input seconds is 7200 seconds i.e. 2 hours, I cannot allow time to be just (2), because then I wouldn’t know the unit of that 2 (2 days? 2 hours? ..). With the above technique to tag the time value with its unit (inspired from format-seconds), the time value will be set as ((2 . "h")) instead. That way, it would read clearly as 2 hours, 0 minutes, and 0 seconds.
Back inside gen-time-string, I then skip printing the time units that are 0 (unless everything is 0, in which case I print "0s").
```
((and val (> val 0))
 (format "%2d%s " val unit))
(t
 filler) ;`filler' is just white-space
```
So instead of printing "1d 0h 0m 5s", it would print "1d 5s".

Tests #

The test generator did not need to be updated, because the code optimization was completely internal — Return values were not affected.

A code isn’t complete without tests!

As much fun I had writing the above function, I had equal fun in writing its little tester too.

(let* ((rand-bool (lambda()
 "(random 2) will return either 1 or 0, so
 frac will be either t or nil"
 (= 1 (random 2))))
 (count 0)
 (secs '(0 1 60 61
 3600 3601 3660 3661
 86400 86401 86460 86461
 90000 90001 90060 90061))
 (len-secs (length secs))
 (secs-rand1 (mapcar (lambda (s)
 (let ((add-sec (funcall rand-bool))
 (add-min (funcall rand-bool))
 (add-hr (funcall rand-bool))
 (add-day (funcall rand-bool)))
 (when add-sec
 (setq s (+ s 1)))
 (when add-min
 (setq s (+ s 60)))
 (when add-hr
 (setq s (+ s (* 60 60))))
 (when add-day
 (setq s (+ s (* 60 60 24))))
 s))
 secs))
 secs-rand2)
 (dotimes (_ (* 2 len-secs))
 (let* ((frac (funcall rand-bool))
 (sec (if frac
 (/ (random 100000000) 100.00)
 (random 1000000))))
 (push sec secs-rand2)))
 (dolist (sec (append secs secs-rand1 secs-rand2))
 (message "%9.2f seconds → %s" sec (modi/seconds-to-human-time sec))
 (cl-incf count)
 (when (= 0 (mod count len-secs))
 (message (make-string 40 ?─)))))

Code Snippet 2: Test Generator

The test also makes use of a let-bound lambda, for the rand-bool function which I use to randomly return t or nil.
The secs list is a set of directed tests, in which the day, hour, minute and second units in time get set to 1 in all possible combinations. (If you are into binary numbers, think of 0000, 0001, .. up to 1111.)
The secs-rand1 is a partly randomized version of secs where one or more of the above time units would get randomly added by 1.
The secs-rand2 is a totally randomized list of time in seconds where the time could be anywhere in the [0, 1000000) range, fractional times with 2 decimal places included.

Test Output #

Upon evaluating both Code Snippet 1 and Code Snippet 2, you will get an output like below:

 0.00 seconds → 0s
 1.00 seconds → 1s
 60.00 seconds → 1m
 61.00 seconds → 1m 1s
 3600.00 seconds → 1h
 3601.00 seconds → 1h 1s
 3660.00 seconds → 1h 1m
 3661.00 seconds → 1h 1m 1s
 86400.00 seconds → 1d
 86401.00 seconds → 1d 1s
 86460.00 seconds → 1d 1m
 86461.00 seconds → 1d 1m 1s
 90000.00 seconds → 1d 1h
 90001.00 seconds → 1d 1h 1s
 90060.00 seconds → 1d 1h 1m
 90061.00 seconds → 1d 1h 1m 1s
────────────────────────────────────────
 60.00 seconds → 1m
 86402.00 seconds → 1d 2s
 86521.00 seconds → 1d 2m 1s
 3722.00 seconds → 1h 2m 2s
 3661.00 seconds → 1h 1m 1s
 90061.00 seconds → 1d 1h 1m 1s
 90121.00 seconds → 1d 1h 2m 1s
 93662.00 seconds → 1d 2h 1m 2s
172861.00 seconds → 2d 1m 1s
176462.00 seconds → 2d 1h 1m 2s
176520.00 seconds → 2d 1h 2m
 86521.00 seconds → 1d 2m 1s
176460.00 seconds → 2d 1h 1m
 90062.00 seconds → 1d 1h 1m 2s
 93660.00 seconds → 1d 2h 1m
 93661.00 seconds → 1d 2h 1m 1s
────────────────────────────────────────
429733.00 seconds → 4d 23h 22m 13s
902957.30 seconds → 10d 10h 49m 17.30s
684313.07 seconds → 7d 22h 5m 13.07s
 62058.42 seconds → 17h 14m 18.42s
799077.55 seconds → 9d 5h 57m 57.55s
347952.39 seconds → 4d 39m 12.39s
 31041.30 seconds → 8h 37m 21.30s
242839.97 seconds → 2d 19h 27m 19.97s
852518.67 seconds → 9d 20h 48m 38.67s
160038.24 seconds → 1d 20h 27m 18.24s
689297.00 seconds → 7d 23h 28m 17s
 64048.00 seconds → 17h 47m 28s
870956.98 seconds → 10d 1h 55m 56.98s
608767.00 seconds → 7d 1h 6m 7s
167796.00 seconds → 1d 22h 36m 36s
114940.07 seconds → 1d 7h 55m 40.07s
────────────────────────────────────────
106163.46 seconds → 1d 5h 29m 23.46s
701980.00 seconds → 8d 2h 59m 40s
258706.73 seconds → 2d 23h 51m 46.73s
 33609.98 seconds → 9h 20m 9.98s
639774.63 seconds → 7d 9h 42m 54.63s
338533.00 seconds → 3d 22h 2m 13s
365910.00 seconds → 4d 5h 38m 30s
140002.00 seconds → 1d 14h 53m 22s
365024.20 seconds → 4d 5h 23m 44.20s
497072.00 seconds → 5d 18h 4m 32s
304307.67 seconds → 3d 12h 31m 47.67s
337126.00 seconds → 3d 21h 38m 46s
711862.00 seconds → 8d 5h 44m 22s
746474.22 seconds → 8d 15h 21m 14.22s
200503.00 seconds → 2d 7h 41m 43s
952391.00 seconds → 11d 33m 11s
────────────────────────────────────────

Source #

You can find the latest version of this code at seconds-to-human-time.el (first revision).

Closing #

Your car will be ready in 2h 13m 20s, and the simulation took 15h 3m 47.90s in CPU time.

Figure 1: Screenshot of seconds-to-human-time.el in Emacs

Appendix #

Test output using `format-seconds` #

Instead of using (modi/seconds-to-human-time sec) in the test generator, if I use the below form using format-seconds to get as close as to what I want:

(message "%9.2f seconds → %s"
 sec
 (replace-regexp-in-string
 " days?" "d"
 (replace-regexp-in-string
 " hours?" "h"
 (replace-regexp-in-string
 " minutes?" "m"
 (replace-regexp-in-string
 " seconds?" "s"
 (format-seconds "%2D %2H %2M %z%2S" sec))))))

I get the output on the left below. For brevity, I have pasted only few snippets of the whole test for comparison:

 0.00 seconds → 0s 0.00 seconds → 0s
 1.00 seconds → 1s 1.00 seconds → 1s
 60.00 seconds → 1m 0s 60.00 seconds → 1m
 61.00 seconds → 1m 1s 61.00 seconds → 1m 1s
 3600.00 seconds → 1h 0m 0s 3600.00 seconds → 1h

 3661.00 seconds → 1h 1m 1s 3661.00 seconds → 1h 1m 1s
 86400.00 seconds → 1d 0h 0m 0s 86400.00 seconds → 1d
 86401.00 seconds → 1d 0h 0m 1s 86401.00 seconds → 1d 1s
 86460.00 seconds → 1d 0h 1m 0s 86460.00 seconds → 1d 1m
 86461.00 seconds → 1d 0h 1m 1s 86461.00 seconds → 1d 1m 1s
 90000.00 seconds → 1d 1h 0m 0s 90000.00 seconds → 1d 1h
 90001.00 seconds → 1d 1h 0m 1s 90001.00 seconds → 1d 1h 1s
 90060.00 seconds → 1d 1h 1m 0s 90060.00 seconds → 1d 1h 1m
 90061.00 seconds → 1d 1h 1m 1s 90061.00 seconds → 1d 1h 1m 1s

# Below the random numbers are different on both sides, but the thing to note is the loss
# fractional values (on the left) when seconds are not integers.

288128.50 seconds → 3d 8h 2m 8s 902957.30 seconds → 10d 10h 49m 17.30s
989679.28 seconds → 11d 10h 54m 39s 684313.07 seconds → 7d 22h 5m 13.07s
803137.00 seconds → 9d 7h 5m 37s 347952.39 seconds → 4d 39m 12.39s
 39361.00 seconds → 10h 56m 1s 689297.00 seconds → 7d 23h 28m 17s

Code Snippet 3: Using format-seconds (Left), using modi/seconds-to-human-time (Right)

Notice the redundant 0h, 0m, 0s on the left, and also the loss of seconds precision (the latter point is not a big deal though).

Code (Revision 1) #

Here’s the code, and notes about that follow after that:

(defun modi/seconds-to-human-time (&optional seconds)
 "Convert SECONDS to \"DDd HHh MMm SSs\" string.

SECONDS is a non-negative integer or fractional number.

SECONDS can also be a list of such numbers, which is the case
when this function is called recursively.

When called interactively, if a region is selected SECONDS is
extracted from that, else the user is prompted to enter those."
 (interactive)
 (let ((inter (called-interactively-p 'interactive)))
 (when inter
 (let ((seconds-str (if (use-region-p)
 (buffer-substring-no-properties (region-beginning) (region-end))
 (read-string "Enter seconds: "))))
 (setq seconds (string-to-number seconds-str)))) ;"1" -> 1, "1.2" -> 1.2, "" -> 0
 (let* ((MINUTE 60)
 (HOUR (* 60 MINUTE))
 (DAY (* 24 HOUR))
 (sec (cond
 ((listp seconds) ;This is entered only by recursive calls
 (car (last seconds)))
 ((and (numberp seconds) ;This is entered only in the first entry
 (>= seconds 0))
 seconds)
 (t
 (user-error "Invalid argument %S" seconds))))
 (gen-time-string
 (lambda (time inter)
 "Return string representation of TIME.
TIME is of the type (DD HH MM SS), where each of those elements
are numbers. If INTER is non-nil, echo the time string in a
well-formatted manner instead of returning it."
 (let* ((rev-time (reverse time))
 (sec (nth 0 rev-time))
 (min (nth 1 rev-time))
 (hr (nth 2 rev-time))
 (day (nth 3 rev-time))
 (filler " ")
 (sec-str (cond
 ((> sec 0)
 (if (integerp sec)
 (format "%2ds" sec)
 (format "%5.2fs" sec)))
 ((and (= sec 0) (null min) (null hr) (null day)) ;0 seconds
 " 0s")))
 (min-str (if (and min (> min 0))
 (format "%2dm " min)
 filler))
 (hr-str (if (and hr (> hr 0))
 (format "%2dh " hr)
 filler))
 (day-str (if (and day (> day 0))
 (format "%2dd " day)
 filler))
 (str (string-trim-right
 (concat day-str hr-str min-str sec-str))))
 (if inter
 (message "%0.2f seconds → %s"
 seconds
 (string-trim (replace-regexp-in-string " +" " " str)))
 str))))
 (time (cond
 ((>= sec DAY) ;> day
 (let* ((days (/ (floor sec) DAY))
 (rem (- sec (* days DAY))))
 (cond
 ((= rem 0)
 (list days 0 0 0))
 ((< rem MINUTE)
 ;; Note that (list rem) instead of just `rem' is being
 ;; passed to the recursive call to
 ;; `modi/seconds-to-human-time'. This helps us
 ;; distinguish between direct and re-entrant calls to
 ;; this function.
 (append (list days 0 0) (modi/seconds-to-human-time (list rem))))
 ((< rem HOUR)
 (append (list days 0) (modi/seconds-to-human-time (list rem))))
 (t
 (append (list days) (modi/seconds-to-human-time (list rem)))))))
 ((>= sec HOUR) ;> hour AND < day
 (let* ((hours (/ (floor sec) HOUR))
 (rem (- sec (* hours HOUR))))
 (cond
 ((= rem 0)
 (list hours 0 0))
 ((< rem MINUTE)
 (append (list hours 0) (modi/seconds-to-human-time (list rem))))
 (t
 (append (list hours) (modi/seconds-to-human-time (list rem)))))))
 ((>= sec MINUTE) ;> minute AND < hour
 (let* ((mins (/ (floor sec) MINUTE))
 (rem (- sec (* mins MINUTE))))
 (cond
 ((= rem 0)
 (list mins 0))
 (t
 (append (list mins) (modi/seconds-to-human-time (list rem)))))))
 (t ;< minute
 (list sec)))))
 ;; If `seconds' is a number and not a list, this is *not* a recursive
 ;; call. Return the time as a string only then. For re-entrant
 ;; executions, return the `time' list instead.
 (if (numberp seconds)
 (funcall gen-time-string time inter)
 time))))

Code Snippet 4: Seconds to Human Time (Revision 1)

Most of this snippet is just the day/hour/minute/second math. Apart from that, here are some points that I found of interest:

I did not always want to prompt the user to enter the input argument. If a region was selected, the function assumes that the user selected a number, and skips the prompt step. So I used a plain (interactive) form instead of using (interactive "sPrompt: ") or (interactive "r"). See (eintr) Interactive Options and (elisp) Interactive Codes to learn about interactive and its codes.
Instead of in-lining a modular chunk of logic, like the one where I convert a list like (1 2 3 4) into "1d 2h 3m 4s", I assigned it to a let-bound symbol gen-time-string. That allowed the logic to be more discernible when used in:
```
(if (numberp seconds)
 (funcall gen-time-string time inter)
 time)
```
- Also interesting is the fact that these let-bound lambdas can have their own doc-strings too.
I make use of recursion in this function! But I needed this function to return a string (using that gen-time-string function) only when all the nested calls to itself were returned. So to distinguish between a direct call to the function, and re-entrant calls, when doing the latter, I make the input number a list of that number.
- So while the function might take an input number like 7 for a direct call, that same number, when needed to call to a recursive call, would get passed as (list 7) or '(7).
- If you glance back as that little snippet above, I return the time as a string only if the input seconds is a number — and not a list i.e. only when I am in the “direct call instance”.
The internal variable time is a list and can have up to 4 number elements: (DAYS HOURS MINUTES SECONDS). The key was to always have each of those elements at their respective positions in the list.

If the input seconds is 7200 seconds i.e. 2 hours, I cannot allow time to be just (2), because then I wouldn’t know the unit of that 2 (2 days? 2 hours? ..) — The nested cond logic for setting the time variable ensures it gets set to (2 0 0) instead. That way, it would read clearly as 2 hours, 0 minutes, and 0 seconds.
Back inside gen-time-string, I then skip printing the time units that are 0 (unless everything is 0, in which case I print "0s"). So instead of printing "1d 0h 0m 5s", it would print just "1d 5s".

Converting Org keywords to lower-case

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Fri, 02 Feb 2018 13:42:00 -0500

I never quite liked the trend to have upper-cased keywords in Org documents, like #+TITLE.

So it was a pleasure to see that trend start changing in this Org commit.. so that that same keyword would now be written as #+title.

But now I have quite a few Org documents with the ALL-CAPS keywords and block identifiers. So I came up with this little “lower-case all the Org keywords and block identifiers in the current document” Elisp command:

(defun modi/lower-case-org-keywords ()
 "Lower case Org keywords and block identifiers.

Example: \"#+TITLE\" -> \"#+title\"
 \"#+BEGIN_EXAMPLE\" -> \"#+begin_example\"

Inspiration:
https://code.orgmode.org/bzg/org-mode/commit/13424336a6f30c50952d291e7a82906c1210daf0."
 (interactive)
 (save-excursion
 (goto-char (point-min))
 (let ((case-fold-search nil)
 (count 0))
 ;; Match examples: "#+FOO bar", "#+FOO:", "=#+FOO=", "~#+FOO~",
 ;; "‘#+FOO’", "“#+FOO”", ",#+FOO bar",
 ;; "#+FOO_bar<eol>", "#+FOO<eol>".
 (while (re-search-forward "\\(?1:#\\+[A-Z_]+\\(?:_[[:alpha:]]+\\)*\\)\\(?:[ :=~’”]\\|$\\)" nil :noerror)
 (setq count (1+ count))
 (replace-match (downcase (match-string-no-properties 1)) :fixedcase nil nil 1))
 (message "Lower-cased %d matches" count))))

Here are few examples where M-x modi/lower-case-org-keywords did hundreds of replacements for me, saving me a lot of time 😎.

Source

Narrowing the Author column in Magit

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Mon, 18 Dec 2017 16:36:00 -0500

The Org and Worg git repos can be committed to only via ssh protocol (and not https), and for technical reasons, I cannot commit to those repos via ssh from my primary development machine. So I ended up with a flow that involves pushing commits to those repos using my Nexus 6p phone.

Magit is how I git, and I love to primarily work from the ∗magit-log∗ buffer (M-x magit-status, l b). But the default column widths were not optimal on a Nexus 6p 5.7" screen. And that’s what inspired this tweak, which looks great on a regular desktop monitor too.

Before I jump to the code, you can see in the below figure how the author and commit age columns took up roughly half the screen width on my phone before the tweak (left), and how much easier it is to read the commit messages after the tweak (right).

Figure 1: Org-mode Worg commit log : Before (Left), After (Right)

So the Magit Log columns tweaks are basically:

Abbreviate the authors’ first names to just their initials.
Abbreviate the commit ages to 1-character time units.

To implement those tweaks, I started digging through magit-log.el and found that I would need to customize (i) the magit-log-margin variable and (ii) magit-log-format-margin function.

Customizing magit-log-margin variable allows me to use the abbreviated age instead of the verbose age strings (minute→m, hour→h, day→d, week→w, month→M, year→Y). It’s also where I specify how wide I want the author column to be.. I can now reduce that column width from the default value of 18 to 11 as I am abbreviating the author name too.
Reviewing the magit-log.el code, I realized that I also need to advise the magit-log-format-margin function to implement the author name abbreviation.

In the below code snippet, I am using use-package¹ to customize that variable and function in magit-log.

(use-package magit-log
 :init
 (progn
 ;; (setq magit-log-margin '(t age magit-log-margin-width t 18)) ;Default value
 (setq magit-log-margin '(t age-abbreviated magit-log-margin-width :author 11)))
 :config
 (progn
 ;; Abbreviate author name. I added this so that I can view Magit log without
 ;; too much commit message truncation even on narrow screens (like on phone).
 (defun modi/magit-log--abbreviate-author (&rest args)
 "The first arg is AUTHOR, abbreviate it.
First Last -> F Last
First.Last -> F Last
Last, First -> F Last
First -> First (no change).

It is assumed that the author has only one or two names."
 ;; ARGS -> '((REV AUTHOR DATE))
 ;; (car ARGS) -> '(REV AUTHOR DATE)
 ;; (nth 1 (car ARGS)) -> AUTHOR
 (let* ((author (nth 1 (car args)))
 (author-abbr (if (string-match-p "," author)
 ;; Last, First -> F Last
 (replace-regexp-in-string "\\(.*?\\), *\\(.\\).*" "\\2 \\1" author)
 ;; First Last -> F Last
 (replace-regexp-in-string "\\(.\\).*?[. ]+\\(.*\\)" "\\1 \\2" author))))
 (setf (nth 1 (car args)) author-abbr))
 (car args)) ;'(REV AUTHOR-ABBR DATE)
 (advice-add 'magit-log-format-margin :filter-args #'modi/magit-log--abbreviate-author)))

NOTES

I need to set the magit-log-margin value in the :init block because that variable dynamically sets many other variables in magit-log at the time of loading. So we cannot wait for the whole magit-log to load before setting that variable. I wouldn’t need to worry about this if I were using the Emacs Customize interface to set this variable.
I use modi/magit-log--abbreviate-author to add a :filter-args advice to magit-log-format-margin. The signature of the advised function is (magit-log-format-margin REV AUTHOR DATE) (as of Magit 20180809.1716). So the advice basically replaces the AUTHOR arg with its abbreviated form as explained in the modi/magit-log--abbreviate-author doc-string.

Feel free to ask for more explanation in comments in the case you don’t use use-package. ↩︎

Do Ediff as I mean

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Mon, 09 Mar 2015 17:31:45 -0400

In almost all of my ediff use cases, I would have windows open side-by-side in a frame, and then I would want to do a diff between the two using ediff-buffers.

But emacs doesn’t know that I obviously want to diff those two side-by-side buffers! So it always asks me to select the buffers to be diffed. The same problem is when using ediff-files too.

So I came up with the following helper function to pick the correct ediff command.

(defun modi/ediff-dwim ()
 "Do ediff as I mean.

If a region is active, call `ediff-regions-wordwise'.
Else if the frame has 2 windows with identical major modes,
 - Do `ediff-files' if the buffers are associated to files and the buffers
 have not been modified.
 - Do `ediff-buffers' otherwise.
Else if the current is a file buffer with a VC backend, call `vc-ediff'
Else call `ediff-buffers'."
 (interactive)
 (let* ((num-win (safe-length (window-list)))
 (bufa (get-buffer (buffer-name)))
 (filea (buffer-file-name bufa))
 (modea (with-current-buffer bufa major-mode))
 bufb fileb modeb)
 (save-excursion
 (other-window 1)
 (setq bufb (get-buffer (buffer-name)))
 (setq fileb (buffer-file-name bufb))
 (setq modeb (with-current-buffer bufb major-mode)))
 (cond
 ;; If a region is selected
 ((region-active-p)
 (call-interactively #'ediff-regions-wordwise))
 ;; Else if 2 windows with same major modes
 ((and (= 2 num-win)
 (eq modea modeb))
 (if ;; If either of the buffers is not associated to a file,
 ;; or if either of the buffers is modified
 (or (null filea)
 (null fileb)
 (buffer-modified-p bufa)
 (buffer-modified-p bufb))
 (progn
 (message "Running (ediff-buffers \"%s\" \"%s\") .." bufa bufb)
 (ediff-buffers bufa bufb))
 (progn
 (message "Running (ediff-files \"%s\" \"%s\") .." filea fileb)
 (ediff-files filea fileb))))
 ;; Else if file in current buffer has a vc backend
 ((and filea
 (vc-registered filea))
 (call-interactively #'vc-ediff))
 ;; Else call `ediff-buffers'
 (t
 (call-interactively #'ediff-buffers)))))

Find this function in my emacs config.

My favorite ediff settings are:

;; No separate frame for ediff control buffer
(setq ediff-window-setup-function #'ediff-setup-windows-plain)

;; Split windows horizontally in ediff (instead of vertically)
(setq ediff-split-window-function #'split-window-horizontally)

elisp on A Scripter's Notes

View GitHub Pull Requests in Magit

Locally creating a branch for a PR #

Getting references to all Pull Requests #

Automatically adding PR refs #

Summary #

References #

Emacs Lisp: Advice Combinators

Overview on using advices #

1 :before #

2 :after #

3 :override #

4 :around #

5 :before-while #

6 :before-until #

7 :after-while #

8 :after-until #

9 :filter-args #

10 :filter-return #

Summary #

References #

Org: Show only Post subtree headings

Org Global Cycle #

Skeleton of only Post headings #

The “Collapse All Posts” function #

Binding with C-u C-c TAB #

Result #

Looping through Org mode headings

org-map-entries API #

MATCH strings #

Comparison Types #

Other notes #

Example: Modifying a property in all headings #

org-map-entries References #

Presenting tomelr!

My flow for creating this library #

Adapting the library to fit ox-hugo #

Changes in ox-hugo tests #

What’s next? #

Unblocking some future ox-hugo improvements #

Defining tomelr – A library for converting Lisp expressions to TOML

Using json-encode as reference #

Mapping scalar data to TOML #

About :false #

Mapping lists to TOML #

Mapping lists of lists to TOML #

Mapping other object types #

Closing #

Straight and Curved Quotes in Emacs Lisp

Emacs 25 – Curved quotes rendering feature #

Preventing curved quotes in documentation strings #

Preventing curved quotes in messages #

Summary #

References #

Using Emacs advice to silence messages from functions

Problem Statement #

Course of action #

1 Finding the culprit functions #

Using plain-old grep or ripgrep (rg) #

Using debug-on-message #

2 Figure out how to silence those messages #

3 Applying the ‘silence’ advice #

Limiting the scope of these advices #

Conclusion #

Quick Intro to Emacs Lisp Regression Testing

Test directory #

Test Files #

Feature-specific t***.el test files #

General structure of each feature test #

Wrapper test file all-tests.el #

Makefile #

Summary #

References #

Splitting an Org block into two

Problem #

Action Plan #

Am I in an Org block? #

If so, split the block #

Now make M-return do that #

Full code #

1 `:before` #

2 `:after` #

3 `:override` #

4 `:around` #

5 `:before-while` #

6 `:before-until` #

7 `:after-while` #

8 `:after-until` #

9 `:filter-args` #

10 `:filter-return` #

Binding with `C-u C-c TAB` #

`org-map-entries` API #

`org-map-entries` References #

Adapting the library to fit `ox-hugo` #

Changes in `ox-hugo` tests #

Unblocking some future `ox-hugo` improvements #

Using `json-encode` as reference #

About `:false` #

Using `debug-on-message` #

Feature-specific `t***.el` test files #

Wrapper test file `all-tests.el` #

Now make `M-return` do that #

Test output using `format-seconds` #