I start this post by introducing what the Org mode global cycling command does, what kind of subtree folding I actually need, and then share the solution with code snippets.

Org Global Cycle #

Org mode has a built-in org-global-cycle command that you might be familiar with. It’s bound by default to the S-TAB key. Each time this command is called, the Org buffer will cycle through these states:

1. Overview: Show only the Level 1 headings and collapse everything underneath.
2. Contents: Show only the Org headings and collapse all the content.
3. Show All: Expand all the headings and show their contents too.

If a numeric prefix N is used with this command, it will show only the Org headings up to Level N. For example, C-2 S-TAB will show only the headings up to Level 2.

This is a really helpful command, but I needed something different ..

Skeleton of only Post headings #

I maintain most of this website’s content in a single Org file. I have dozens of blog posts organized in Org subtrees, which I further organize under “category” headings .. It kind of looks like the below mock-up: It’s amazing how many features PlantUML has. If you are interested in creating diagrams like these, check out the PlantUML Salt syntax.

As we can see,

• All the post subtrees are not at Level 1 headings.
• They are also not at a fixed Level N.
• The heading level of the post depends on how many parent categories that post has (and that will also change over time).

I needed to basically show everything leading up to a post subtree heading, and then collapse all the content under that post; even the sub-headings.

The “Collapse All Posts” function #

The modi/org-hugo-collapse-all-posts function defined below meets the above requirement:

1. It first widens the whole buffer and expands all the headings.
2. Then it loops through all the headings and collapses all the post subtrees i.e. all the subtrees that have the EXPORT_FILE_NAME property set. This is where I use the org-map-entries magic.
3. Finally it looks for Org headings that begin with “Footnotes” or “COMMENT” and collapses them as well.

I am using the development version of Org mode (version 9.6, yet to be released as of <2022-06-15 Wed>) which has the new org-fold library. This library obsoletes the use of outline.el library and other code-folding related functions in Org mode. So cl-flet is used to create function symbol aliases that use the org-fold-* functions if available, otherwise they fall back to the legacy functions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

(defun modi/org-hugo-collapse-all-posts ()
  "Collapse all post subtrees in the current buffer.
Also collapse the Footnotes subtree and COMMENT subtrees if
present.

A post subtree is one that has the EXPORT_FILE_NAME property
set."
  (interactive)
  (cl-flet ((show-all (if (fboundp 'org-fold-show-all)
                          #'org-fold-show-all
                        #'org-show-all))
            (hide-subtree (if (fboundp 'org-fold-hide-subtree)
                              #'org-fold-hide-subtree
                            #'outline-hide-subtree)))
    (widen)
    (show-all '(headings))
    ;; Collapse all the post subtrees (ones with EXPORT_FILE_NAME
    ;; property set).
    (org-map-entries #'hide-subtree "EXPORT_FILE_NAME<>\"\"" 'file)
    ;; Also hide Footnotes and comments.
    (save-excursion
      (goto-char (point-min))
      (while (re-search-forward "^\\(\\* Footnotes\\|\\*+ COMMENT\\)"
                                nil :noerror)
        (hide-subtree)))))

Binding with C-u C-c TAB #

The function is ready, but let’s now add a bit of convenience to it.

If a point is under a subtree, C-c TAB will collapse that subtree while showing only Level 1 headings, and if a numeric prefix is used, it will show only those many levels of headings. I decided to bind the above function to C-u C-c TAB because,

1. The behavior of modi/org-hugo-collapse-all-posts falls in the same category as that of C-c TAB.
2. The C-u C-c .. binding rolls off the fingers pretty nicely 😃.

This binding is achieved using one of my favorite Emacs features .. the advice system. The :before-until Advice Combinator is used here, which means that if the advising function (below) returns a nil, the advised or the original function org-ctrl-c-tab is not called.

The advising function below detects if the C-u prefix argument is used. If it is, the modi/org-hugo-collapse-all-posts function is called, otherwise the original org-ctrl-c-tab function is called.

1
2
3
4
5
6
7
8
9

(defun modi/org-ctrl-c-tab-advice (&rest args)
  "Run `modi/org-hugo-collapse-all-posts' when
doing \\[universal-argument] \\[org-ctrl-c-tab]."
  (let ((do-not-run-orig-fn (equal '(4) current-prefix-arg)))
    (when do-not-run-orig-fn
      (modi/org-hugo-collapse-all-posts))
    do-not-run-orig-fn))

(advice-add 'org-ctrl-c-tab :before-until #'modi/org-ctrl-c-tab-advice)

Result #

After evaluating the above two snippets, when I do C-u C-c TAB in my “blog posts” Org buffer, I see this:

It matches that earlier mockup — Mission accomplished! 💯

In my previous HTML5 Validator post, I mentioned:

I was a bit disappointed to see validation errors on my site, but then it wasn’t too bad .. 46 errors.

But they truly say ..

So .. once I realized that my site had 46 validation errors, I lost that bliss .. and I couldn’t rest easy — I had to fix them all 😁.

This post summarizes the categories of those errors and how I fixed The fixes mentioned in this post refer to changes in Org mode content. But you should be able to derive equivalent fixes for Markdown too. them all.

1 Avoid duplicate heading id attributes #

"file:/public/notes/​string-fns-nim-vs-python/index.html":636.34-636.46: error: Duplicate ID "notes".
"file:/public/notes/​nim-fmt/index.html":300.34-300.52: error: Duplicate ID "older-issue".
"file:/public/notes/​nim-fmt/index.html":306.20-306.33: error: Duplicate ID "floats".
"file:/public/page/6/index.html":29.39-29.54: error: Duplicate ID "fnref:1".

Errors with above kind of signatures were fixed by,

1. Converting headings to description lists

   I had a bunch of generic headings like “Notes” and “Older Issue” in some of my posts. After taking a second look at those, it made more sense to convert those to description lists. So in Org mode, I converted headings like * Notes to description lists - Notes ::.

2. Setting CUSTOM_ID heading property

   For the cases, where the headings needed to be left as so, their IDs were uniquified by setting their CUSTOM_ID property. For example, below fixed the Duplicate ID “floats” errors.

    ** Precision
    ..
    *** Floats
   +:PROPERTIES:
   +:CUSTOM_ID: precision-floats
   +:END:
    ..
    ** Type (only for numbers)
    ..
    *** Floats
   +:PROPERTIES:
   +:CUSTOM_ID: type-floats
   +:END:
   

   Code Snippet 1: Using CUSTOM_ID property to uniquify heading ID's

3. Prevent footnote links in post summaries

   This issue was due to me not being conscious about how the footnote references work in a post versus on a page outside that post’s context. The issue was caused by footnote references getting into the post summaries parsed by Hugo, which will then show up on the list pages.

   The fix was simple — Edit the post summaries so that they don’t contain any footnote references.

2 Remove inline <style> elements #

"file:/public/​grep-po/index.html":51.139-51.145: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/​how-do-i-write-org-mode/index.html":23.194-23.200: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)

Errors with above kind of signatures were fixed by,

1. Avoiding export of raw <style> elements in the Markdown content

   I figured out which functions were responsible for injecting <style> elements in Markdown content and then advised them to stop that. After applying these advises, I lost the in-content rules for CSS classes .org-center and .csl-entry. So I put those rules directly in this website’s CSS.

   (defun modi/org-blackfriday-center-block (_center-block contents info)
     (let* ((class "org-center"))
       (format "<div class=\"%s\">%s\n\n%s\n</div>"
               class (org-blackfriday--extra-div-hack info) contents)))
   (advice-add 'org-blackfriday-center-block :override #'modi/org-blackfriday-center-block)
   
   (defun modi/org-cite-csl-render-bibliography (bib-str)
     (replace-regexp-in-string "<style>\\.csl-entry[^<]+</style>" "" bib-str))
   (advice-add 'org-cite-csl-render-bibliography :filter-return #'modi/org-cite-csl-render-bibliography)
   

   Code Snippet 2: Emacs-Lisp advices to prevent <style> elements in exports

2. Removing unnecessary micro-styling

   I found a single case, where an inline CSS rule was defined in content for CSS class .repr-type for a table. I just removed that without affecting the looks of that rendered table too much.

3 Ensure that all images have captions or alt attributes #

"file:/public/​hugo-use-goat-code-blocks-for-ascii-diagrams/index.html":24.130-24.255: error: An "img" element must have an "alt" attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.

Errors with above kind of signatures were easily fixed by ensuring that all images had captions Thankfully, there were only two images that were missing captions. . The Hugo figure shortcode adds the caption to the alt attribute if the alt is not specified separately.

As an example, here’s how I fixed the above error:

+ #+name: fig__disproportionate_box_drawing
+ #+caption: Disproportionate box drawing characters
[[file:images/​hugo-use-goat-code-blocks-for-ascii-diagrams/ascii-diagram-rendered-in-plain-text-code-block.png]]

4 Do not have hyperlinks in headings #

"file:/public/​using-emacs-advice-to-silence-messages-from-functions/index.html":151.687-151.732: error: Start tag "a" seen but an element of the same type was already open.
"file:/public/​using-emacs-advice-to-silence-messages-from-functions/index.html":151.748-151.751: error: Stray end tag "a".
"file:/public/​auto-count-100daystooffload-posts/index.html":114.446-114.475: error: Start tag "a" seen but an element of the same type was already open.
"file:/public/​auto-count-100daystooffload-posts/index.html":114.446-114.475: error: End tag "a" violates nesting rules.
"file:/public/​auto-count-100daystooffload-posts/index.html":114.526-114.529: error: Stray end tag "a".

While the hyperlinks in headings work well, they created invalid HTML in the Hugo-generated TOC. So while these errors were created technically because of a bug in Hugo It’s really cool when you end up finding a bug in an upstream project while trying to fix the errors in your own thing 😎. , I wanted to fix these errors on my end as soon as I can.

I reviewed the errors, and this is all it took to get rid of them all:

1. Remove manually inserting hyperlinks in headings

    show two methods of finding sources of any printed messages.
   -***** Using plain-old /grep/ or [[https://github.com/BurntSushi/ripgrep][/rg/]]
   +***** Using plain-old /grep/ or /ripgrep (rg)/
    This method is pretty easy (but not robust) to use if the search
    ..
    Org source directory and search for the /"org-babel-exp process .."/
   -string ..
   +string using [[https://github.com/BurntSushi/ripgrep][~rg~]] ..
   

   Code Snippet 4: Removing hyperlink from a heading

2. Remove Org Radio links that created links in headings

   Here, a Org heading happened to contain the string “Day count”, which was also an an Org Radio link in that post. While ideally that shouldn’t have mattered, I removed that radio link to get around this Hugo bug.

    .. /Just may be/. But regardless, I am already enjoying writing once
   -again, and it's great to see the <<<Day count>>> (counting up to 100)
   +again, and it's great to see the Day count (counting up to 100)
    increase with each new post!
   

   Code Snippet 5: Removing an Org Radio link

Validation Ignores #

Above fixes fixed 43 out of 46 errors, but the remaining 3 were unfixable.

5 Ignore errors due to hyperlinks in inline SVG #

"file:/public/notes/​plantuml/index.html":114.474-114.678: error: Attribute "title" not allowed on element "a" at this point.

This error was caused by hyperlinks in inline SVG elements. These SVG elements are created by PlantUML. The hyperlinks in SVG feature works great, and as these are generated by PlantUML, I chose to just ignore these errors.

I ignored this error by adding the --ignore-re 'notes/plantuml.*Attribute.*title.*not allowed' switch to the html5validator command.

6 Ignore files not expected to serve HTML content #

"file:/public/googleFOO.html":1.1-1.52: error: Non-space characters found without seeing a doctype first. Expected "<!DOCTYPE html>".

The googleFOO.html file here is not a valid HTML file. It’s a just a cookie file that was used by Google to verify that I own this domain.

This error was masked by adding the --ignore 'googleFOO' switch to the html5validator command.

Summary #

Once I fixed the 43 errors by tweaking the Org mode content, and added those two ignores, I had zero validation errors! 🎉

If you are interested in the fix details, here are the commits.

Below question on Mastodon by the user @postroutine inspired me to write this post:

I got a lot of Org-Mode headings and I want to modify their properties (add, remove, edit). Is there a function to do the same modifications on each heading?

I think that the best solution to that question is using the org-map-entries function.

But somehow when replying to that question then, that wasn’t what first came to my mind! .. when ironically that function is the main function that enables my preferred subtree-based flow in ox-hugo 😆. So I am writing this post to better ingrain the following concept in myself ..

Next,

1. I will give a give introduction to the org-map-entries API.
2. Then provide a super-short solution to the above question.

org-map-entries API #

I will give only a broad level overview on how to use this function. I would encourage the reader to refer to the resources at the end of this post to learn more about it.

So let’s start by looking at this function’s signature:

(org-map-entries FUNC &optional MATCH SCOPE &rest SKIP)

The org-map-entries function iterates through all the headings meeting the MATCH criteria in the determined SCOPE, and then calls the specified function FUNC at each of those headings.

• The FUNC function accepts no arguments and is called at the beginning of each Org heading.
• The optional second argument MATCH is either nil, t or a search string.
  • If MATCH is nil or t, all headings will be visited by the iteration and FUNC will be called on all of them.
  • But if MATCH is a string, the headings will first be filtered based on that string and then the FUNC will be called on only those.
• For explanations on the optional SCOPE and SKIP arguments, see Org Info: Using the Mapping API or C-h f org-map-entries from within Emacs.

Here’s a typical org-map-entries call that loops through all the headings in the visible buffer: (org-map-entries #'some-function) where all the optional argument values are nil. Next, we’ll see some examples of string-type MATCH arguments used for filtering the headings.

MATCH strings #

Below table shows few examples of match string patterns.

Comparison Types #

• If the comparison value is a plain number, a numerical comparison is done, and the allowed operators are ‘<’, ‘=​’, ‘>’, ‘<=​’, ‘>=​’, and ‘<>’.
• If the comparison value is enclosed in double quotes, a string comparison is done, and the same operators are allowed.
• If the comparison value is enclosed in curly braces, a regexp match is performed. For this comparison, only ‘=​’ (regexp matches) and ‘<>’ (regexp does not match) operators are allowed.
• Comparison with dates and Group Tags is also possible. See Org Info: Matching tags and properties for more details.

Other notes #

• The property names are case-insensitive. So these all work the same: "COLOR<>\"blue\"", "color<>\"blue\"", "Color<>\"blue\"".
• The “tag” and “property” matches can be mixed up using the boolean ‘&’, ‘|’, ‘+’ and ‘-​’ operators. So searching ‘+LEVEL=3+boss-TODO​="DONE"’ lists all level three headlines that have the tag ‘boss’ and are not marked with the TODO keyword ‘DONE’.
• ‘&’ binds more strongly than ‘|’.
• Grouping of match expressions using parentheses is not supported.

Example: Modifying a property in all headings #

Below is an example solution to the Mastoson question that I referenced in the beginning of this post.

(defun test/set-property-at-heading ()
  "Function to be called at the beginning of an Org heading."
  (let ((el (org-element-at-point)))
    (org-set-property "foo" (org-element-property :title el))))
(org-map-entries #'test/set-property-at-heading)

• It defines a function that parses the Org element at point using org-element-at-point, gets the title property of the element This function is designed to be called by org-map-entries and so the point at the time of calling this function will always be on a heading. , and sets that to the headline element’s foo property.
• The org-map-entries call now simply calls this function on each heading in the visible scope of the Org buffer.

org-map-entries References #

• C-h f org-map-entries
• Org Info: Using the Mapping API
• Org Info: Matching tags and properties

Most of my blog posts are mainly to serve as documentation for my future self This post will serve to remind me how to get the :LOGBOOK: notes working once again in case I end up with some issue there. . So when I get a chance, I try to fix outdated stuff in my old blog posts. And along with the act of updating things, adding brief notes describing those updates comes naturally to me.

Prior forms of adding post updates #

As I author my posts in Org mode, I can easy enter a date stamp using the org-time-stamp command (bound by default to C-c . RET) and follow that by the update note.

While that worked, that approach bothered me because those notes didn’t have consistent format across multiple posts. For example, I might type the update as “*Update (<time stamp>)*: <note>” in one post, while I might type the same in a description list form in another: “- <time stamp>) :: <note>”.

To solve the consistency problem, I came up with this Org macro:

#+macro: update - $1 :: $2

This worked mostly … except when the update text needed to be a bit longer, like a paragraph. It didn’t look elegant in that case. Also, if the text had a comma character in there, it needed to be escaped with a backslash (\).

Introducing :LOGBOOK: #

So when I learned A little bit of history .. I learned about the :LOGBOOK: drawer when Adam Porter mentioned it in ox-hugo Issue # 203 back in September 2018. I wanted to use that feature, but I didn’t have time and/or know-how on how exactly I would parse those Org Drawers in ox-hugo until very recently (May 2022)! about the Org :LOGBOOK: drawer, it solved all those problems: (i) consistency in adding notes (ii) easy to add update notes – in fact much easier (iii) easy to type long form notes (iv) no comma escaping needed.

Org Drawers look like this:

Content before the drawer
:DRAWERNAME:
Content inside the drawer
:END:
Content after the drawer

and they can be inserted anywhere in your Org content using the org-insert-drawer command (bound by default to C-c C-x d).

:LOGBOOK: is a special kind of drawer that’s auto-inserted by Org mode when certain actions are detected, like changing the TODO state of a subtree, or adding a note . That latter action is what this blog post is about.

Adding notes to :LOGBOOK: #

You need to enable this feature using one of these methods:

1. Set the org-log-into-drawer variable to a non-nil value (typically t) in your Emacs config, or as a file-local variable, or in your project’s .dir-locals.el (⭐ my preference).
2. Set #+startup: logdrawer to enable this for the whole Org file.
3. To enable this feature for only selected subtrees, set the :LOG_INTO_DRAWER: t property in the subtree (or one of its parent subtrees).

Once this is set, call the org-add-note command (bound by default to C-c C-z). That will open a window with ∗Org Note∗ buffer where-in you will type your post update and then C-c C-c to save it to the subtree’s :LOGBOOK: drawer. If that drawer didn’t exist already, it will be created directly under the subtree’s heading.

The note will get recorded in this format by default under the current subtree:

* Subtree title
:LOGBOOK:
- Note taken on <current date and time> \\
  <note text>
:END:

As you see, you only type the note text, and the time-stamp is inserted automatically.

:LOGBOOK: Notes Example #

Here are the update notes from one of my posts:

:LOGBOOK:
- Note taken on <2018-08-26 Sun> \\
  Mention =org-babel-demarcate-block=, tweak the =org-meta-return= advice.
- Note taken on <2018-08-23 Thu> \\
  Use ~M-return~ instead of ~C-return~ for splitting blocks and
  support upper-case blocks (though I [[* Converting Org keywords to lower-case][don't prefer those]]!).
:END:

You can see how they rendered at the top of the Splitting an Org block into two post.

If you are an ox-hugo user following the subtree-based export flow, and would like to export :LOGBOOK: notes in a similar fashion, check out the ox-hugo Manual: Drawers page for details.

References #

• Org Info: Drawers
• Org Info: Tracking TODO state changes

I am assuming that you already know what Org mode is and that’s why you are here 😃.

You would want to build Org from its development branch (main branch) to get the latest and greatest goodies plus bug fixes! Go through the ORG-NEWS file to read about what’s new in the main branch.

If you like what you see there, here are the steps for installing the development version of Org.

1 Clone the Org repo #

git clone https://git.savannah.gnu.org/git/emacs/org-mode.git

2 Build Setup #

1. Copy <REPO_ROOT>/mk/default.mk to <REPO_ROOT>/local.mk
2. Tweak local.mk (optional)

Customizing local.mk (Optional) #

Here are few variables that you might like to change in the local.mk:

prefix

Org installation directory

prefix = /dir/where/you/want/to/install/org # Default: /usr/share

The .el files will go to $(prefix)/emacs/site-lisp/org by default. If you’d like to change that, change the lispdir variable too.

infodir

Org Info installation directory. I like to keep the Info file for development version of Org in a separate directory.

infodir = $(prefix)/org/info # Default: $(prefix)/info

ORG_MAKE_DOC

Types of Org documentation you’d like to build by default. Setting below generates only the Org Info manual.

ORG_MAKE_DOC = info # Default: html pdf

3 Build #

make autoloads
make
make doc
make install

Type make help for help on the Org Makefile. Type make helpall to get a detailed help, or see the Org build system help.

4 Set the correct paths in your Emacs config #

1. Update load-path to remove the Org version that ships with Emacs. Do the same if you have Org installed via GNU ELPA If you need the latest stable version of Org mode, install it from GNU ELPA. too.
2. Also remove the associated old Org mode Info manuals from Info-directory-list.
3. Update the load-path and Info-directory-list variables to point to the Org mode source code and Info manual built using the main branch.

Below code does all that but make sure this code is executed after you do (package-initialize), but before you require the org package . You can use use-package and make sure that this order of code evaluation is always correct — Just put the code from Code Snippet 2 where <HERE> is shown in the below snippet:

(use-package org
  :preface
  <HERE>)

(defvar modi/org-version-select 'dev
  "Variable to choose the version of Org to be loaded.
Valid values are `dev', `elpa' and `emacs'.

When set to `dev', the development version of Org built locally
is loaded.
When set to `elpa', Org is installed and loaded from GNU ELPA.
When set to `emacs', the Org version shipped with Emacs is used.")

(defvar modi/default-lisp-directory "/your/emacs/share/dir/version/lisp/"
  "Directory containing lisp files for the Emacs installation.

This value must match the path to the lisp/ directory of your
Emacs installation.

If Emacs is installed using --prefix=\"${PREFIX_DIR}\" this value
would typically be
\"${PREFIX_DIR}/share/emacs/<VERSION>/lisp/\".")

(defvar org-dev-lisp-directory "/value/of/lispdir/in/local.mk"
  "Directory containing lisp files for dev version of Org.

This value must match the `lispdir' variable in the Org local.mk.
By default the value is \"$prefix/emacs/site-lisp/org\", where
`prefix' must match that in local.mk too.")

(defvar org-dev-info-directory "/value/of/infodir/in/local.mk"
  "Directory containing Info manual file for dev version of Org.

This value must match the `infodir' variable in the Org local.mk.")

(when (and org-dev-lisp-directory
           org-dev-info-directory)
  (with-eval-after-load 'package
    ;; If `modi/org-version-select' is *not* `emacs', remove the Emacs
    ;; version of Org from the `load-path'.
    (unless (eq modi/org-version-select 'emacs)
      ;; Remove Org that ships with Emacs from the `load-path'.
      (let ((default-org-path (expand-file-name "org" modi/default-lisp-directory)))
        (setq load-path (delete default-org-path load-path))))

    ;; If `modi/org-version-select' is *not* `elpa', remove the Elpa
    ;; version of Org from the `load-path'.
    (unless (eq modi/org-version-select 'elpa)
      (dolist (org-elpa-install-path (directory-files-recursively
                                      package-user-dir
                                      "\\`org-[0-9.]+\\'"
                                      :include-directories))
        (setq load-path (delete org-elpa-install-path load-path))
        ;; Also ensure that the associated path is removed from Info
        ;; search list.
        (setq Info-directory-list (delete org-elpa-install-path Info-directory-list))))

    (let ((dev-org-path (directory-file-name org-dev-lisp-directory))
          (dev-org-info (directory-file-name org-dev-info-directory)))
      (if (eq modi/org-version-select 'dev)
          (progn
            (add-to-list 'load-path dev-org-path)
            ;; It's possible that `org-dev-info-directory' is set to
            ;; an unconventional value, in which case, it will not be
            ;; automatically added to `Info-directory-alist'. So add
            ;; it to `Info-directory-alist' manually.
            (add-to-list 'Info-directory-list dev-org-info))

        ;; If `modi/org-version-select' is *not* `dev', remove the
        ;; development version of Org from the `load-path', and its
        ;; Info from the Info search list.
        (setq load-path (delete dev-org-path load-path))
        (with-eval-after-load 'info
          (setq Info-directory-list (delete dev-org-info Info-directory-list)))))))

Note

Remember that you need to correctly set the values of these 3 variables in the above snippet:

• modi/default-lisp-directory
• org-dev-lisp-directory
• org-dev-info-directory

5 Testing that the right Org version got loaded #

1. Restart Emacs (Don’t be lazy — do it!)
2. M-x org-version – That should show something like this in the echo area:

   Org mode version 9.5.3 (release_9.5.3-482-gd5a52b @ /home/kmodi/usr_local/apps/7/emacs/emacs-28/share/emacs/site-lisp/org/)
   

   This message format is broken down as:

   Org mode version <ORG-VERSION> (release_<ORG-VERSION>-NNN-g<GIT-HASH> @ <PREFIX>/emacs/site-lisp/org/)
   

If the GIT-HASH and PREFIX above are what you expect, congratulations! You did it! 🎉

Else, let me know in comments if I can help you.

Future Org development version updates #

1. Below will do git pull and build Org.

   make update
   

2. Restart Emacs.

References #

• setup-org.el in my Emacs config
• setup-packages.el in my Emacs config

In my previous post Defining tomelr, I started toying with the idea of creating a library that would help convert any Lisp expression to a TOML config, and I share my vision (specification) of what this library would look like.

I wasn’t even sure if I would be able to make the tomelr library feature-complete at least to the extent of what ox-hugo was already doing! But to my surprise, the library development snowballed to a completion much earlier than I thought, and additionally it helped fix some inconsistencies that the older TOML generation code had in ox-hugo.

In this post, I start by (i) giving a broad overview of how the development of tomelr happened, then (ii) briefly describe how it got integrated into ox-hugo, and finally (iii) how the use of this library will unblock the path to addition of some cool features to ox-hugo.

My flow for creating this library #

1. Write the spec for the library.
   • List all the formats of Lisp data I would expect it to process.
   • List the corresponding TOML data I would expect it to generate.
   • Ensure that I am not inventing my own lisp syntax by confirming that the expected TOML output matches the JSON generated from that same lisp form (using the Emacs built-in json.el library).
2. That helped me write the tests first! – Test Driven Development (TDD).
3. I started with writing tests for TOML booleans and then implementing that (because that was the simplest and easiest). Of course, I used ert for this! ert helped me quickly create small modular tests Here’s the ert test for booleans as an example. and efficiently iterate through modifications in the library code until I got the tests to pass.
4. Once that got working, I set up a continuous integration system using GitHub Actions (GHA). I used GHA because I host my library on GitHub. Also I already have a tried and tested setup that I could get up and going in a matter of few seconds. In general, this concept would apply to any Continuous Integration system. The CI setup step should come early in the development of any project so that incremental feature additions don’t start breaking previously added features 😃.
5. The library development just snowballed after this point .. added support for integers, floats, regular strings, multi-line strings, arrays, TOML tables, arrays of TOML tables. By this time, the library was about 80% finished.
6. Then came the difficult part .. stabilizing the library to support all the varieties of Lisp data I can think of. I must have put in double the time spent so far to finish the remaining 20% of the planned features for this library 👉 tomelr test suite
7. Once I had the test suite complete and passing, it was time to do some code cleanup:
   • Remove duplicate code and break them off into smaller helper functions.
   • See if the function defined in this library is already defined somewhere else (in this case, I was able to use json-plist-p directly from json.el).
   • Proof read the code.
   • Proof read the docstrings and run M-x checkdoc to fix their formatting.
   • Ensure that the code compiles without any warnings.
   • Remove unnecessary customization options and case statements from the library (while continuously ensuring that the ert tests still pass).

Adapting the library to fit ox-hugo #

After polishing the library by its stand-alone testing, I decided to use it with ox-hugo and see how the test suite in that repo fared.

Of course I saw that a lot of tests failed now 😁.

The main issue was that tomelr was constructing multi-line strings such that the spaces translated exactly from Lisp data to TOML. So (tomelr-encode '((foo . "line1\nline2"))) would generate:

foo = """
line1
line2"""

whereas ox-hugo expected the same TOML to look like:

foo = """
  line1
  line2
  """

I had intentionally decided for ox-hugo to have this latter format for multi-line strings because (i) it made it more readable with the triple-quotes out of the way on their own lines, (ii) the indented lines prevented the multi-line string from getting mixed with surrounding TOML parameters, and most importantly (iii) these strings were processed by the Hugo Markdown parser, and so it wasn’t sensitive to horizontal spaces.

And so the tomelr-indent-multi-line-strings feature was born (commit) which optionally made tomelr export multi-line strings as expected by ox-hugo 😎.

Changes in ox-hugo tests #

Once I had finalized the integration of tomelr into ox-hugo, I had only about 30 tests change out of roughly 400 tests. These changes were welcome as they fixed all the inconsistencies in the older TOML generation code in ox-hugo. If interested, you can see this commit for the diff and details, but here’s the gist:

1. Now nil value of a key in Lisp consistently implies that the key should not be exported to TOML. So '((foo . nil)) will result in foo not getting exported to TOML, whether that’s a top-level key or a key in a nested TOML map or array. If you need to set a key to a boolean false, use "false" or any value from tomelr-false.
2. Earlier empty string value as in '((foo . "")) behaved like the current nil implementation. That’s not the case any more. Now that empty string will export as foo = "" in TOML.
3. Now if a string has a quote character (") in it, that value will auto-export as TOML multi-line string. I like the readability of this more than that of backslash-escaped double-quotes.
4. Now the nested tables like [menu."nested menu"] export with their parent table keys like [menu]. As per the TOML spec, this is not required. But now that tomelr has added a generic support for any TOML table, this change happens as a result of consistency 💯.

In summary, the changes in ox-hugo TOML front-matter exports were mostly cosmetic, and if they were not cosmetic, they were consistency fixes.

What’s next? #

tomelr

The library is pretty much feature complete ✨ as many of the examples from TOML v1.0.0 spec have been added to its test suite, and .. it is supporting all the ox-hugo use cases.

The library though has one limitation that I’d like to resolve at some point — Right now, we require the Lisp data to first list all the scalar keys and then list the TOML tables and arrays of tables. But at the moment, I don’t know how to fix that, and also ox-hugo is not affected by that (because it already populates the front-matter alist in the correct order). So fixing this is not urgent, but of course, if someone can help me out with that, I’d welcome that! 🙏.

ox-hugo

Given that tomelr allows robustly exporting any Lisp data expression to TOML, I do not see any value in continuing with YAML generation support using the old custom code.

📢 In near future, I plan to get rid of the org-hugo-front-matter-format customization variable from ox-hugo — thus deprecating YAML export support This change should not functionally affect the YAML front-matter fans out there because the front-matter that ox-hugo is exporting is mainly for Hugo’s consumption. The only scenario where I see that this change can be breaking is if the user is using YAML format extra front-matter blocks. If so, unfortunately, they will need to convert those to TOML manually. and sticking with using just TOML for the front-matter.

Unblocking some future ox-hugo improvements #

This decision will open up the doors to add more features to ox-hugo like:

1. Exporting Org :LOGBOOK: drawers to TOML front-matter (ox-hugo # 504)

2. Exporting Org Special Blocks to user-configurable front-matter (ox-hugo # 627)

3. Supporting more complex data in Lisp form using :EXPORT_HUGO_CUSTOM_FRONT_MATTER: which could translate to nested TOML tables or arrays of TOML tables.

   Finally, there won’t be a need to use the “Extra front-matter” workaround. For example, it would be possible to represent the data in that first example on that page as :foo ((:bar 1 :zoo "abc") (:bar 2 :zoo "def")) in the :EXPORT_HUGO_CUSTOM_FRONT_MATTER: property.

After writing my previous post, I had shared its link on the Org mode mailing list For folks who don’t know, you can email the Org mode mailing list (emacs-orgmode@gnu.org) about pretty much anything Org mode related: any help you want, a bug you want to report, a patch you want to submit, or just share your positive or negative experiences with Org mode or thank someone for their awesome work. here, and received some helpful feedback from Max Nikulin.

This post is a response to his feedback, and it also describes a few improvements made for info: link exports in ox-hugo based on that.

💪¹ 1: Exported links now point to separate pages in the manual #

Feedback

  For <info:emacs#Browse-URL> export to html produces the following link:
      https://www.gnu.org/software/emacs/manual/html_mono/emacs.html#Browse_002dURL
  I think, a better variant is
      https://www.gnu.org/software/emacs/manual/html_node/emacs/Browse_002dURL.html
  even though for the Org manual I often prefer single-page HTML version.

I really liked this suggestion!

• Earlier, the Info node references were exporting to an anchor on a huge single-page HTML manual e.g. emacs.html#Browse_002dURL
• After implementing this fix, the exported link points to a separate HTML page for that node e.g. emacs/Browse_002dURL.html.

Now, [[info:emacs#Screen]] exports and renders to Emacs Info: Screen, and a Top level node like [[info:emacs#Top]] exports and renders to Emacs Info.

For consistency, all info: links will now be exported to URLs pointing to the separate node HTML pages, even for Org manual links.

💪 2: Export Org manual links to orgmode.org URLs #

Feedback

(About an example info: link pointing to Org manual: [[info:org#Top]])

And the link target ideally should be https://orgmode.org/manual/index.html

This was another great suggestion!

By default, the Org manual info: links will export to URLs pointing to the Org manual shipped with Emacs. That manual will be associated with the Org mode version that shipped with the last stable version of Emacs. So depending on how long it has been since the last Emacs release, you could be referring to quite an older version of the Org manual.

If you install Org from GNU Elpa, you can get an update of the latest stable Org version every week ! There’s a related post on Org mode’s Release Flow that might interest you. And the manual hosted on https://orgmode.org gets updated every week as well. I would like to always refer to the latest Org manual with the latest fixes and documentation improvements, and so I agree with Max’s suggestion above.

Now [[info:org#Top]] exports and renders as Org Info and the Org manual nodes like [[info:org#Working with Source Code]] also export as a link to a page in that manual: Org Info: Working with Source Code.

💪 3: Link hover text shows the elisp code for accessing the Info node #

Feedback

  I would prefer
      info "(org) Top"
  to
      Org Info: Top
  since the former may be pasted to M-x : or to shell command prompt.

I wanted the link descriptions in the exported markdown to be more “English” and understandable to people not familiar with the Emacs Info interface. So I decided to keep this mostly unchanged ..

I did not change the link description, but I did add a new HTML title attribute to the link. This attribute contains the Emacs Lisp code needed to access the Info manual from Emacs. The user will see the text from this attribute when they hover the mouse over the links. If you haven’t already discovered this feature, try hovering the mouse over that last link above and you should see this 😃 :

Of course, the user will still not be able to copy that text and paste in Emacs. But it will still provide them enough hint on how to access Info nodes in Emacs.

Final code in ox-hugo that exports the info: links #

Here’s the version of the org-hugo--org-info-export function that handles the exports of info: links as of today (<2022-04-15 Fri>):

888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912

(defun org-hugo--org-info-export (path desc format)
  "Add support for exporting [[info:..]] links for `hugo' format."
  (let* ((parts (split-string path "#\\|::"))
         (manual (car parts))
         (node (or (nth 1 parts) "Top"))
         (title (format "Emacs Lisp: (info \\\"(%s) %s\\\")" manual node))
         (desc (or desc
                   (if (string= node "Top")
                       (format "%s Info" (capitalize manual))
                     (format "%s Info: %s" (capitalize manual) node))))
         ;; `link' below is mostly derived from the code in
         ;; `org-info-map-html-url'.
         (link (cond ((member manual org-info-emacs-documents)
                      (let ((manual-url (if (string= (downcase manual) "org")
                                            "https://orgmode.org/manual"
                                          (format "https://www.gnu.org/software/emacs/manual/html_node/%s" manual)))
                            (node-url (if (string= node "Top")
                                          "index.html"
                                        (concat (org-info--expand-node-name node) ".html"))))
                        (format "%s/%s" manual-url node-url)))
                     ((cdr (assoc manual org-info-other-documents)))
                     (t
                      (concat manual ".html")))))
    (when (member format '(md hugo))
      (format "[%s](%s \"%s\")" desc link title))))

Link to code in the repo

I will leave it as an exercise for the reader to map the lines in above code snippet to the features described in this post 🍀.

This post was inspired out of the exercise of support info: Org link exports in ox-hugo.

There are 3 steps:

1. Storing a link from any buffer that you want to later document in an Org file.
2. Inserting that link that you stored earlier.
3. Exporting the Org document.

I am focusing on the info: Org links in this post, they these steps apply to any Org link type.

1 Storing links to Info nodes #

If you follow the instructions on Org Info: Activation, you’ll see that it is recommended This post assumes that you have actually set this binding. If you haven’t yet, go do it already! (global-set-key (kbd "C-c l") #'org-store-link) to bind the org-store-link command to C-c l in your Emacs global map.

org-store-link stores an Org Mode compatible link based on the context i.e. type of buffer where the point is. Org supports a variety of links as documented in Org Info: External Links.

If the point is in an ∗Info∗ buffer and org-store-link is called, it stores an info: type link that will lead back to the same node where the point was.

Let’s say you have opened the Org Info manual and you hit C-c l. You should then see this in the Echo Area:

Stored: org#Top

The link to Org manual’s Top node is stored by that command, but only temporarily – it isn’t yet saved anywhere.

2 Inserting stored links #

These stored links can now be saved in an Org file. To do that,

• Visit an Org file or a buffer and call M-x org-insert-link (bound to C-c C-l by default) and follow the prompts.
• An immediate RET after the prompt will select the last stored link to be inserted.
• The second prompt allows you to change the link description, and then the link will be inserted.

If I insert the org#Top Info link that I stored earlier over here, and accept the default link description, it will paste as [[info:org#Top][org#Top]].

But .. how did the link description become “org#Top” by default?

Org Link library for info: links (ol-info) #

The default description for info: type links was set at the time of storing those links, with the help of the org-info-store-link function in ol-info.el. That same description is then suggested at the time of inserting that link.

This library defines these “actions” for [[info:..]] type of links:

:store

Defines when (in which buffer, major mode, etc.) the “store” operation should store to an info: type link, and what metadata should be stored. Some of the stored metadata are: the Info file name, node name, link string and description string.

:follow

Defines what action to perform when org-open-at-point (C-c C-o) is called with point on the link. Hitting this binding with point on an info: link will open or switch to that ∗Info∗ node.

:export

Defines how the link should be exported for various backends. ol-info defines how the info: links should be exported, but only for html and texinfo backends.

3 Exporting [[info:..]] links using ox-hugo #

ol-info.el defines the :export “action” such that info: links get converted to hyperlinks pointing to online Org manual pages, when exporting using ox-html.

But it didn’t do the same when exporting using ox-hugo. That feature got added recently in this ox-hugo commit. Now ox-hugo exports [[info:org#Top][org#Top]] to org#Top.

I didn’t like the # that org-info-store-link added to the default description (notice that previous link). So I added a tiny little feature to the info: export behavior 😁 — If you leave the info: link descriptions empty when inserting the links, ox-hugo injects its own “auto description” which are better (in my opinion).

Now when that same Info link is inserted description-less as [[info:org#Top]], it renders to: Org Info.

Summary #

1. Store link (from anywhere in Emacs) : C-c l
2. Insert link (in an Org file) : C-c C-l
3. Export the Org file as usual.

The main focus of my previous post was the CSS for sidenotes. I also showed the companion HTML wrapper needed for that CSS to work. Here it is again for convenience:

<span class="sidenote-number">
  <small class="sidenote">
    sidenote content
  </small>
</span>

Hugo shortcode #

Now .. You might be thinking that it’s not too practical to type that thing for each new sidenote. And you are right! That’s where we use the Hugo Shortcode feature. Hugo Shortcodes are like Org Macros that are typically used to insert some HTML template around the user entered content. If you look at the above snippet, the outer span and small tags are just HTML boilerplate that the user will need to type mechanically around each sidenote. So the shortcodes are perfect for solving this annoyance.

If we design a paired shortcode for this, the above HTML example will change to below:

{{% sidenote %}}
sidenote content
{{% /sidenote %}}

As you can see, this shortcode This is a markdown-rendering shortcode i.e. a shortcode with {{​% .. %}} instead of {{​< .. >}}. So the insides of this shortcode will passed on to the Hugo Markdown renderer. now enables the user to focus on content writing than on writing boilerplate HTML. And the definition of that shortcode is pretty simple as well:

<span class="sidenote-number"><small class="sidenote">{{ .Inner }}</small></span>

The only Hugo-templating syntax used here is {{ .Inner }}. You can read more about it here, but the tl;dr is that the .Inner holds whatever content the user entered inside the {{​% sidenote %}} and {{​% /sidenote %}} tags.

Once you save that shortcode snippet to a layouts/sidenote.html file in your Hugo site’s directory, you are ready to use it.

Org mode special block #

It would be a shame to populate the Org source of this blog with Hugo-specific shortcodes when I am using ox-hugo 😄.

So instead, we just use the plain-old Org special blocks and tell ox-hugo that this “sidenote” special block be exported as a markdown-rendering Hugo shortcode. I do that by just placing this at the top of my Org file See the Hugo Paired Shortcodes section in the ox-hugo manual to learn more about this. :

#+hugo_paired_shortcodes: %sidenote

.. and then writing a “sidenote” type Org special block:

abc
#+begin_sidenote
example sidenote content
#+end_sidenote
def

Normally, Org mode would insert a paragraph break where any Org block (like that sidenote special block) is used. So above would render as:

But I am pretty sure that nobody would render sidenote references in their main text as above—instead, below is more normal 😄.

In that normal example of the sidenote, ox-hugo trims the whitespace around the sidenote block. That is configured by customizing the org-hugo-special-block-type-properties variable:

(with-eval-after-load 'ox-hugo
  (add-to-list 'org-hugo-special-block-type-properties '("sidenote" . (:trim-pre t :trim-post t))))

You can read more about this whitespace trimming feature in the ox-hugo manual.

Summary #

One-time configuration #

1. Create and save a “sidenote” Hugo shortcode as shown here.
2. Configure ox-hugo to export “sidenote” special block as a markdown-rendering shortcode.
3. Configure whitespace to be trimmed around those blocks using org-hugo-special-block-type-properties customization variable.

Org special block: sidenote #

Now just use the #+begin_sidenote .. #+end_sidenote Org special blocks wherever you need sidenotes.

I ventured out to start writing about a 100+ line Emacs Lisp snippet in my config, and then I thought — Wouldn’t it be nice if I can quickly split out that huge snippet into smaller Org Src blocks?

And so this blog post happened.

Problem #

If I have a huge Org Src block, I’d like to split it into multiple Org Src blocks so that I can write my explanations in-between.

So I’d like to quickly split up:

#+begin_src emacs-lisp
(message "one")
(message "two")
#+end_src

into:

#+begin_src emacs-lisp
(message "one")
#+end_src

#+begin_src emacs-lisp
(message "two")
#+end_src

.. like this.

☝ Click for animation.

Action Plan #

1. Write a function to return non-nil if point is in any Org block – Not just “src”, “example”, “export” or any of the inbuilt Org blocks.. but also any Org Special block like #+begin_foo .. #+end_foo.
2. Write a function that does this imagined block splitting.
3. Overload the M-return binding so that this block splitting function gets called only when the point is inside an Org block (detected using that first function).

Am I in an Org block? #

Before venturing into writing this function, I looked at these existing ones, but none did what I exactly wanted:

org-in-src-block-p

Returns non-nil only if the point is in a #+begin_src .. #+end_src block; not when point is in any other Org block.

org-in-block-p

Returns non-nil only if the point is in one of the pre-defined block names passed as a list ('("src" "example" "quote" ..)). So this again won’t work as I cannot pre-define all Org Special blocks.

So I define the below modi/org-in-any-block-p function that returns non-nil if the point is in-between any #+begin_FOOBAR .. #+end_FOOBAR. Thankfully, I was able to reuse a lot of logic from the org-between-regexps-p function (org-in-block-p uses that function internally).

(defun modi/org-in-any-block-p ()
  "Return non-nil if the point is in any Org block.

The Org block can be *any*: src, example, verse, etc., even any
Org Special block.

This function is heavily adapted from `org-between-regexps-p'."
  (save-match-data
    (let ((pos (point))
          (case-fold-search t)
          (block-begin-re "^[[:blank:]]*#\\+begin_\\(?1:.+?\\)\\(?: .*\\)*$")
          (limit-up (save-excursion (outline-previous-heading)))
          (limit-down (save-excursion (outline-next-heading)))
          beg end)
      (save-excursion
        ;; Point is on a block when on BLOCK-BEGIN-RE or if
        ;; BLOCK-BEGIN-RE can be found before it...
        (and (or (org-in-regexp block-begin-re)
                 (re-search-backward block-begin-re limit-up :noerror))
             (setq beg (match-beginning 0))
             ;; ... and BLOCK-END-RE after it...
             (let ((block-end-re (concat "^[[:blank:]]*#\\+end_"
                                         (match-string-no-properties 1)
                                         "\\( .*\\)*$")))
               (goto-char (match-end 0))
               (re-search-forward block-end-re limit-down :noerror))
             (> (setq end (match-end 0)) pos)
             ;; ... without another BLOCK-BEGIN-RE in-between.
             (goto-char (match-beginning 0))
             (not (re-search-backward block-begin-re (1+ beg) :noerror))
             ;; Return value.
             (cons beg end))))))

• (case-fold-search t) ensures that either #+BEGIN_ .. or #+begin_ .. match.
• The regular expression in block-begin-re matches with "#+begin_src foo" or " #+begin_src foo" or "#+BEGIN_EXAMPLE" or "#+begin_FOOBAR" or ..
• The limit-up and limit-down are set to the buffer locations of the previous and next Org headings. The following regexp searches are limited to happen in those bounds for better performance.
• The block-end-re is dynamically constructed based on the string matched using block-begin-re. This is so that if "#+begin_quote" is found initially, it matches the block ending with specifically "#+end_quote" and not something like "#+end_src".
• nil is returned if the point is not between #+begin_FOOBAR .. #+end_FOOBAR.

Caveat

I haven’t gone extra lengths to support nested block cases, specifically where the point is outside the inner-most block, but still inside the outer block:

#+begin_src org
▮
#+begin_src emacs-lisp
(message "hello!")
#+end_src
#+end_src

If so, split the block #

With the “point in an Org block” detection working, I now needed the split to happen with these rules:

1. If the point is anywhere on the line, but not at the beginning of the line (BOL),

   • Go to the end of the line, and then split the block.

     So if the point¹ is after the first message identifier, or at the end of that first message line:

     #+begin_src emacs-lisp
     (message "one")▮
     (message "two")
     #+end_src
     

     Split the block at the point after (message "one") and move the point to between the split blocks:

     #+begin_src emacs-lisp
     (message "one")
     #+end_src
     ▮
     #+begin_src emacs-lisp
     (message "two")
     #+end_src
     

2. Otherwise (if point is at BOL),

   • Split the block exactly at that point.

     So if the point is at the beginning of the second message line:

     #+begin_src emacs-lisp
     (message "one")
     ▮(message "two")
     #+end_src
     

     Split the block at the point before (message "two") and move the point to between the split blocks:

     #+begin_src emacs-lisp
     (message "one")
     #+end_src
     ▮
     #+begin_src emacs-lisp
     (message "two")
     #+end_src
     

So here’s the code that follows that spec:

(defun modi/org-split-block ()
  "Sensibly split the current Org block at point."
  (interactive)
  (if (modi/org-in-any-block-p)
      (save-match-data
        (save-restriction
          (widen)
          (let ((case-fold-search t)
                (at-bol (bolp))
                block-start
                block-end)
            (save-excursion
              (re-search-backward "^\\(?1:[[:blank:]]*#\\+begin_.+?\\)\\(?: .*\\)*$" nil nil 1)
              (setq block-start (match-string-no-properties 0))
              (setq block-end (replace-regexp-in-string
                               "begin_" "end_" ;Replaces "begin_" with "end_", "BEGIN_" with "END_"
                               (match-string-no-properties 1))))
            ;; Go to the end of current line, if not at the BOL
            (unless at-bol
              (end-of-line 1))
            (insert (concat (if at-bol "" "\n")
                            block-end
                            "\n\n"
                            block-start
                            (if at-bol "\n" "")))
            ;; Go to the line before the inserted "#+begin_ .." line
            (beginning-of-line (if at-bol -1 0)))))
    (message "Point is not in an Org block")))

• The regexp for extracting block-start is the same as block-begin-re in Code Snippet 1, but with different sub-grouping.
• The block-end string is derived from sub-group 1 of block-start string – just replacing “begin_” with “end_”.
• And then based on if the point was initially at BOL (at-bol), the insertion of newlines and movement of point is done accordingly.

Now make M-return do that #

With these two functions evaluated, M-x modi/org-split-block will work right away.

I needed to have the Org block splitting happen with an intuitive binding — like M-return.

• By default, M-return is used to either create new headings, or do other things like insert an item, wrap a region in table, etc. based on the context. See the doc-string of org-meta-return (function bound to this key by default) for more info.
• But it doesn’t have a context for “point in an Org block”. So it tries to create a heading when inside a block too, which doesn’t make much sense.
• So fix that by adding that context.

So I advise org-meta-return to call modi/org-split-block when the point is inside an Org block.

The advising function modi/org-meta-return is the same as the advised function org-meta-return (as of <2018-08-26 Sun>), except that a new context (modi/org-in-any-block-p) is added.

You can tweak the precedence of this new context by moving the ((modi/org-in-any-block-p) #'modi/org-split-block) form in that cond form.

(defun modi/org-meta-return (&optional arg)
  "Insert a new heading or wrap a region in a table.

Calls `org-insert-heading', `org-insert-item',
`org-table-wrap-region', or `modi/org-split-block' depending on
context.  When called with an argument, unconditionally call
`org-insert-heading'."
  (interactive "P")
  (org-check-before-invisible-edit 'insert)
  (or (run-hook-with-args-until-success 'org-metareturn-hook)
      (call-interactively (cond (arg #'org-insert-heading)
                                ((org-at-table-p) #'org-table-wrap-region)
                                ((org-in-item-p) #'org-insert-item)
                                ((modi/org-in-any-block-p) #'modi/org-split-block)
                                (t #'org-insert-heading)))))
(advice-add 'org-meta-return :override #'modi/org-meta-return)

Now with the point in any Org block, M-return away!

Full code #

Look for the source of modi/org-split-block (and dependent functions) added to setup-org.el in my Emacs config.

A super-quick intro to Org mode #

Org mode is a fantastic major mode for Emacs, and people use it for all sorts of things like keeping notes, maintaining TODO lists, writing documentation, or even blogging—like I am doing here.

Inconsistency #

Every now and then, I would see “Org mode” and related phrases written in the “wild” (like blogs, Reddit posts, tweets) as Org-mode, org-manual, org file, etc., with a mix-and-match of cases and hyphens.

So here is an attempt to familiarize more people with the documentation standard for referring to “Org” stuff. Below I am quoting the text from the official Org Documentation Standards:

• Prefer “Org mode” to “Org-mode” or “org-mode”. This is simply because it reflects an existing convention in The Emacs Manual which consistently documents mode names in this form - “Text mode”, “Outline mode”, “Mail mode”, etc.
• Likewise refer, if at all possible, to “Org file or “Org buffer” meaning with, great generality, any file or buffer which requires use of some part of Org to edit it properly.
• Org uses “org-…” to ring fence a name space for itself in the Emacs code base. This is obviously retained in code snippets.

The right way #

Only in the Elisp code, is it called org-mode, because it is a major mode, and has to be named so in code by convention, and for its separate org-* name space.

But for blogging, documentation, etc. –

Reference #

• Org mode Documentation Standard

I have often seen questions and confusion about why certain fixes or features added to Org mode did not show up in its update on GNU Elpa.

Below flow chart is an attempt to answer all such questions, and also my first attempt at creating one using PlantUML (legacy syntax) See the legacy vs new (beta) PlantUML syntax for activity diagrams. When I tried the new (beta) syntax, it did not allow using the “labels”.. see the as bugfix, as main syntax in the flowchart source code. .

(*) --> "My Org commit"

--> "Discuss on Org mailing list"

if "Bug fix commit?" then
  -->[yes] "Commit to Org **bugfix**" as bugfix
else
  ->[no] if "**bugfix** compatible doc fix commit?" then
    -->[yes] bugfix
  else
    ->[no] "Commit to Org **main**" as main
  endif
endif

bugfix --> "Merge to Org **main**"
  --> main
bugfix --> "Wait till next Monday"
  --> "Push to GNU Elpa" as push
bugfix --> "Short maturity time"
  --> "Cut a minor release" as minor_release

minor_release --> ===RELEASE===
minor_release --> main

main --> "**Long maturity time**"
  --> "Cut a major release" as major_release

major_release --> ===RELEASE===
major_release --> "Merge to Org **bugfix**"
  --> bugfix

===RELEASE=== --> push
===RELEASE=== --> "Squash and commit to Emacs release or main branch"
  --> (*)

push --> (*)

If you have a simple Org table like:

| 0.5 |
| 1.5 |
|-----|
|     |

And you want the A3 cell to contain the sum of A1 and A2 cells, you would add this at the bottom on that table and evaluate it (C-c C-c).

#+tblfm: @>$1=vsum(@1..@>>)

That formula reads as: “Set the value of the cell is last row (@>)¹, first column ($1) be equal to the sum of all cells in the same column from row 1 (@1) to second-to-the-last row (@>>).”

But then you end up with an odd-looking 2. instead of 2.0 in the result cell:

| 0.5 |
| 1.5 |
|-----|
|  2. |
#+tblfm: @>$1=vsum(@1..@>>)

So I had posted a question on the Org mailing list to understand if this was a bug — it was not.

Using printf style %0.1f formatter #

Thanks to the reply from Thierry Banel to that question, one of the solutions is to use field formatters, like in printf in C (and many other languages):

| 0.5 |
| 1.5 |
|-----|
| 2.0 |
#+tblfm: @>$1=vsum(@1..@>>);%0.1f

Using Calc f1 formatter #

Another solution, also from Thierry, was to use the Calc f1 formatter instead of %0.1f (and similarly f5 instead of %0.5f).

| 0.5 |
| 1.5 |
|-----|
| 2.0 |
#+tblfm: @>$1=vsum(@1..@>>);f1

Note

While the %.1f format is handy for those who are used to printf() syntax, note that Calc unlimited precision numbers are converted to double floats before applying %.1f. Whereas f1 operates on Calc numbers without conversion.

• 1 Using MathJax
• 2 Using HTML + CSS
  • Define these macros in Org
  • CSS
  • Use the Org macros
• References

Ever wondered how to show L^aT_eX in HTML or a Hugo blog post exported from Org?

There are 2 ways to do this:

1. Using MathJax – \(\LaTeX\).
2. Using HTML + CSS – L^aT_eX.

1 Using MathJax #

If you don’t mind including the MathJax script, it’s as simple as typing $\LaTeX$ in Org, which results in \(\LaTeX\).

Similarly \(\TeX\) ($\TeX$) also works, though not $\XeTeX$.

2 Using HTML + CSS #

And here’s another way if you don’t want to include MathJax.

Define these macros in Org #

#+macro: tex @@html:<span class="tex">T<sub>e</sub>X</span>@@
#+macro: latex @@html:<span class="latex">L<sup>a</sup>T<sub>e</sub>X</span>@@
#+macro: xetex @@html:<span class="xetex">X<sub>&#398;</sub>T<sub>E</sub>X</span>@@

CSS #

Add this CSS directly in the page within a #+begin_export html / #+end_export block, or add that CSS your site’s stylesheet.

<style>
.tex, .latex, .tex sub, .latex sub, .xetex sub {
  font-size: 1em;
}

.tex sub, .latex sub, .latex sup, .xetex sub {
  text-transform: uppercase;
}

.tex sub, .latex sub, .xetex sub {
  vertical-align: -0.5ex;
  margin-left: -0.1667em;
  margin-right: -0.125em;
}

.latex sup {
  font-size: 0.85em;
  vertical-align: 0.15em;
  margin-left: -0.36em;
  margin-right: -0.15em;
}
</style>

Use the Org macros #

- {{{tex}}}
- {{{latex}}}
- {{{xetex}}}

Export that from Org to HTML or Hugo, and you get:

• T_eX
• L^aT_eX
• X_ƎT_EX

References #

• stackoverflow.com
• tess.oconnor.cx
• hroy.eu

So it was a pleasure to see that trend start changing in this Org commit.. so that that same keyword would now be written as #+title.

But now I have quite a few Org documents with the ALL-CAPS keywords and block identifiers. So I came up with this little “lower-case all the Org keywords and block identifiers in the current document” Elisp command:

(defun modi/lower-case-org-keywords ()
  "Lower case Org keywords and block identifiers.

Example: \"#+TITLE\" -> \"#+title\"
         \"#+BEGIN_EXAMPLE\" -> \"#+begin_example\"

Inspiration:
https://code.orgmode.org/bzg/org-mode/commit/13424336a6f30c50952d291e7a82906c1210daf0."
  (interactive)
  (save-excursion
    (goto-char (point-min))
    (let ((case-fold-search nil)
          (count 0))
      ;; Match examples: "#+FOO bar", "#+FOO:", "=#+FOO=", "~#+FOO~",
      ;;                 "‘#+FOO’", "“#+FOO”", ",#+FOO bar",
      ;;                 "#+FOO_bar<eol>", "#+FOO<eol>".
      (while (re-search-forward "\\(?1:#\\+[A-Z_]+\\(?:_[[:alpha:]]+\\)*\\)\\(?:[ :=~’”]\\|$\\)" nil :noerror)
        (setq count (1+ count))
        (replace-match (downcase (match-string-no-properties 1)) :fixedcase nil nil 1))
      (message "Lower-cased %d matches" count))))

Here are few examples where M-x modi/lower-case-org-keywords did hundreds of replacements for me, saving me a lot of time 😎.

Source

Update (2017/06/16)

Updated the github source references as per the changes in hugo v0.23¹. Also made the shell script ShellCheck-passing! 🙌 §

You already got go installed, and that’s why are you reading it.

         If not – Go! Install it!

So.. now that you have go installed, why settle for the release version of hugo! You are reading this blog post generated using the bleeding edge of hugo².

In addition, Hugo v0.19 onwards supports the org-mode syntax³, which is so much more awesome than markdown, especially if you are using emacs.

This post is about how you can install the latest versions of hugo and the go package that adds the org-mode support – goorgeous.

First time install of hugo #

When any package gets installed using go get, it gets installed inside the $GOPATH/src/ directory. We install hugo using go get for this one-and-only time to get that correct directory structure in there:

go get -u -v github.com/gohugoio/hugo

Installing hugo from master #

Now you should find the hugo source code in $GOPATH/src/github.com/gohugoio/hugo/.

From here on, run all commands in that directory! ⁴

To get the latest files from the master branch do:

git fetch --all # fetch new branch names if any
git checkout master
# I also do hard reset as I don't develop any code in go
git reset --hard origin/master

Govendor #

Hugo does its package dependency management using govendor, which, not surprisingly, is also a go package. (goorgeous is one of these dependent packages.)

So you would install govendor like any other go package:

go get -u -v github.com/kardianos/govendor

The package dependency database is stored in $GOPATH/src/github.com/gohugoio/hugo/vendor/vendor.json. This JSON file specifies what other go packages need to be installed from which git repo, using which commit.

Govendor makes installing the right versions of the dependent packages easy – Just run the below.

govendor sync

Latest goorgeous #

Note: If you do not wish to update to the latest version of goorgeous, you can skip this step.

The commit hash for goorgeous in that JSON file might not point to its latest version. But we are interested in getting the latest-and-greatest org-mode support for hugo ..

The good news is that govendor allows that:

# Fetch the goorgeous package from its master branch
govendor fetch github.com/chaseadamsio/goorgeous

See the Govendor Quick Start Guide for frequently used commands.

Static Executable #

We want to make the hugo executable a static executable.

That way it is 100% portable — independent of dynamically linked libraries. This is also how the Continuous Integration engines (like the Gitlab CI Runner that generates this blog) can use this custom built hugo binary without any dynamic library dependency issue.

To enable static building of go binaries, these env variables need to be set:

export CGO_ENABLED=0
export GO_EXTLINK_ENABLED=0

Note that above (and the snippet that follows) work in a bash shell. Even if your default shell is not bash, you can run the full bash script at the end of this post.

Building Hugo #

We now finally build hugo, with some customization so that running hugo version prints the Build Date and Commit Hash too.

package="github.com/gohugoio/hugo"
commithash=$(git rev-parse --short HEAD 2>/dev/null)
builddate=$(date +%FT%T%z)
go install -v \
   -ldflags "-X ${package}/hugolib.CommitHash=${commithash} \
             -X ${package}/hugolib.BuildDate=${builddate}" \
   ${package}

This will install the binary as $GOPATH/bin/hugo.

Now don’t forget to add $GOPATH/bin to your $PATH!

Verify that the binary got built as expected by running hugo version.

Hugo Static Site Generator v0.24-DEV-411AC930 linux/amd64 BuildDate: 2017-06-16T11:16:50-04:00

Full Build Script #

Here is the full bash script. You can save it as hugo-build.sh and it will run just fine on any shell (as long as you haven’t removed bash from your system 😄).

#!/usr/bin/env bash

here=$(pwd)

package="github.com/gohugoio/hugo"

export CGO_ENABLED=0
export GO_EXTLINK_ENABLED=0

if ! hash govendor 2>/dev/null
then
    go get -u -v github.com/kardianos/govendor
fi

# Install hugo for the first time so that the ${GOPATH}/src/${package}
# directory gets populated.
if [[ ! -d "${GOPATH}/src/${package}" ]] || ( ! hash hugo 2>/dev/null )
then
    go get -u -v ${package}
fi

# Update to hugo master branch
cd "${GOPATH}/src/${package}" || exit
git fetch --all # fetch new branch names if any
git checkout master
# git fetch --all
# Force update the vendor file in case it got changed
git reset --hard origin/master

# Synchronize all the dependent packages as per the just updated vendor file
govendor sync

# Update the goorgeous package to its master branch
# You can comment out the below line if you do not need to fetch the
# latest version of goorgeous.
govendor fetch github.com/chaseadamsio/goorgeous
# govendor fetch github.com/chaseadamsio/goorgeous@=fixNewlineParagraphs

commithash=$(git rev-parse --short HEAD 2>/dev/null)
builddate=$(date +%FT%T%z)
go install -v \
   -ldflags "-X ${package}/hugolib.CommitHash=${commithash} \
             -X ${package}/hugolib.BuildDate=${builddate}" \
   ${package}

echo "Hugo Version Check:"

cd "${here}" || exit

Full detail is in this reddit post I started.

The fix was to add the following code after the line (funcall lang-mode) in the org-html-fontify-code defun in ox-html.el.

(when (require 'fill-column-indicator nil 'noerror)
    (fci-mode -1))

Make sure you delete ox-html.elc else your patched ox-html.el won’t be effective.

* Section in org mode
Text in org mode. And here is the LaTeX snippet \(\sim 2000k \approx 2^{21}\).

Source

Follow the below steps to create a table ..

1. Start writing the headers of the table starting with a vertical pipe |.

   | Header1 | Header2 | Header3 |▮
   

2. Once you have finished writing the headers, do C-u C-c - to insert a horizontal line above the header row.

   |---------+---------+---------|
   | Header1 | Header2 | Header3 |▮
   

3. Then do C-c - to insert a horizontal line below the header row.

   |---------+---------+---------|
   | Header1 | Header2 | Header3 |▮
   |---------+---------+---------|
   

4. Go down one row ( C-n ) and hit TAB and org-mode will figure out that you need to create a new row and will put the cursor in the first cell of the new row.

   |---------+---------+---------|
   | Header1 | Header2 | Header3 |
   |---------+---------+---------|
   |▮        |         |         |
   

5. You can now use TAB and Shift + TAB to navigate the cells and new rows will be created when you hit TAB when you are in the last cell of the last created row.

   When you want to close the table with a bottom border, hit C-c - when the cursor is in the last row.

   |---------+---------+---------|
   | Header1 | Header2 | Header3 |
   |---------+---------+---------|
   | A       | B       | C▮      |
   |---------+---------+---------|