Posts on A Scripter's Notes

Version controlling Jenkins config

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Wed, 20 Jul 2022 00:18:00 -0400

Jenkins is an amazing free and open source continuous integration and deployment software. But its primary means of configuration is a web UI, and recently that cost me a lot of debug time. That set me down the path of figuring out a way to version control the Jenkins config (the $JENKINS_HOME directory).

Table of Contents

What bit me
.gitignore for Jenkins config
Jenkins Plugin Manager
Full solution

Jenkins is a wonderful piece of software and I use it with my Bitbucket git repos for CI/CD.

Jenkins uses a web UI for its configuration. I dislike that because it’s difficult to document the configuration process without screenshots, and if I need to create a new server, it’s a manual process of clicking through tabs and filling in the text boxes. I didn’t mind this enough to do anything about it .. that is until I finally got bit by it.

What bit me #

Without going into too much detail, that issue was multi-fold:

I had unknowingly messed up the Project-based Matrix Authorization Strategy such that other users in my team were not able to view the Jenkins jobs.
I had also updated the Jenkins server that introduced a bug (JENKINS-68748) where the Test LDAP Settings failed with an error, but the LDAP authentication actually worked!
I had also updated all the plugins after updating Jenkins. So if I rolled back the Jenkins versions, most of the plugins would fail because of incompatibility with the older Jenkins version. I had updated Jenkins after months!

That’s when I wished that my whole Jenkins was version-controlled. That would have allowed me to roll back to the last working “Jenkins image” with the Jenkin version, plugins' versions and my Jenkins config all in sync.

I had delayed doing this because my $JENKINS_HOME was more than 1GB in size and I didn’t have time or motivation to figure out what stuff I should commit and what I should ignore .. But no more — The time had finally come.

`.gitignore` for Jenkins config #

So I did what any good engineer would do .. start looking for a solution online. I found this StackOverflow answer for Is there a way to keep Hudson / Jenkins configuration files in source control?.

That answer shares a .gitignore that ignores files not necessary for configuring a Jenkins server — Example: job builds, workspace, log files, etc. But it didn’t work out of the box because the plugin version info wasn’t getting committed correctly. I had committed everything to git after using the suggested .gitignore and pushed to my git remote. But if I cloned that repo to a different area and attempted to start the Jenkins server from there, it crashed with this message:

2022-07-15 13:07:06.082+0000 [id=31] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading global config
com.thoughtworks.xstream.mapper.CannotResolveClassException: hudson.security.ProjectMatrixAuthorizationStrategy
 at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:81)
 at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
 at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55)
 at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
 at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88)
 at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
 at com.thoughtworks.xstream.mapper.ClassAliasingMapper.realClass(ClassAliasingMapper.java:79)
 at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
 at com.thoughtworks.xstream.mapper.ArrayMapper.realClass(ArrayMapper.java:74)
 at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
 at com.thoughtworks.xstream.mapper.SecurityMapper.realClass(SecurityMapper.java:71)
 at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
 at hudson.util.XStream2$CompatibilityMapper.realClass(XStream2.java:411)
 at hudson.util.xstream.MapperDelegate.realClass(MapperDelegate.java:46)
 at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
 at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:47)
 at hudson.util.RobustReflectionConverter.determineType(RobustReflectionConverter.java:521)
 at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:346)
Caused: jenkins.util.xstream.CriticalXStreamException:
---- Debugging information ----
cause-exception : com.thoughtworks.xstream.mapper.CannotResolveClassException
cause-message : hudson.security.ProjectMatrixAuthorizationStrategy
class : hudson.model.Hudson
required-type : hudson.model.Hudson
converter-type : hudson.util.RobustReflectionConverter
path : /hudson/authorizationStrategy
line number : 12
version : not available
-------------------------------

Code Snippet 1: Snippet of Jenkins crash when attempting to run the server from the freshly cloned git repo

Jenkins Plugin Manager #

So I reached out for help on the Jenkins Community. One of the key contributors to Jenkins, Mark Waite, was tremendously helpful. He suggested using his jenkins-plugin-manager tool. After trying it out for a bit, I realized that this tool had everything I needed for version controlling the plugin versions:

Ability to save a list of installed Jenkins plugins and their versions to a file.
Ability to batch install all the plugins of the versions listed in a file.

This was like doing Python’s plugin management using requirements.txt, except that this was for Jenkins.

Full solution #

With a combination of the .gitignore that I started with from that SO answer, managing plugins using jenkins-plugin-manager, tweaking the .gitignore to my liking, and adding helper Bash scripts for downloading and running Jenkins server binaries, and doing the plugin management, I finally got what I needed:

https://github.com/kaushalmodi/jenkins-minimal

The README on the repo has all the instructions.

Using Git Delta with Magit

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Wed, 06 Jul 2022 22:04:00 -0400

Git Delta is a command line utility that beautifies git diffs in the terminal. But did you know that it can do the same in Magit too?

Table of Contents

Installing delta
Installing magit-delta
Configuring delta

Delta is a highly configurable I am not kidding. Check out the output of delta --help. command line utility that makes the git diffs look better, while also syntax-highlighting Difftastic is another popular diff tool which compares files based on their syntax. I like reviewing git diffs from within Emacs (Magit). But difftastic does not support Magit. the code in the diffs.

When I first heard of “syntax highlighted diffs”, I wasn’t sure what that meant. If you are in the same boat, here’s a screenshot that shows that.

Figure 1: Example of how delta renders a git diff for an ox-hugo commit

But I do most of my git operations including viewing of diffs from within Emacs, using Magit.

.. and thankfully delta works with Magit!

Below screenshot shows how the same diff looks like in Magit.

Figure 2: Example of how magit-delta renders a git diff for an ox-hugo commit

The magit-delta Emacs package makes this possible, which is also developed by the delta author Dan Davison.

Caveat: If the line numbers are enabled in delta, they mess up the interactive expanding and collapsing of diffs in Magit. See Magit Delta Issue # 13 for more details.

Now, I am alright with not seeing the line numbers in Magit. But I really liked to see the line numbers in the side-by-side view in the terminal. Luckily, if disabled the line-numbers feature but enabled the side-by-side view, I got what I wanted!

Line numbers are disabled in Magit and expanding/collapsing of diffs works correctly. I am also really glad that I don’t see the side-by-side view in Magit diffs even when I enable that feature in delta, because I like to have my Emacs buffers only about 90 characters wide.
Line numbers and side-by-side view are enabled in the terminal.

I’ll end this post with pointers to installing delta and magit-delta and how to configure them.

Installing `delta` #

You can install delta (it’s called git-delta in some package managers) using one of the methods listed in its manual, or you can download → extract its statically compiled binary for your OS from its GitHub Releases page.

Installing `magit-delta` #

Once you put this snippet in your Emacs config and evaluate it, it will install this package and enable the magit-delta-mode in the Magit buffers.

(use-package magit-delta
 :ensure t
 :hook (magit-mode . magit-delta-mode))

Code Snippet 1: Installing and enabling magit-delta using use-package

Configuring `delta` #

Here’s a snippet for delta configuration from my .gitconfig. It’s mostly the same as the one in delta’s the Getting Started guide. The main difference in my workaround for the magit-delta issue.

[core]
 pager = delta

[interactive]
 diffFilter = delta --color-only
[add.interactive]
 useBuiltin = false # required for git 2.37.0

[diff]
 colorMoved = default

[delta]
 # https://github.com/dandavison/magit-delta/issues/13
 # line-numbers = true # Don't do this.. messes up diffs in magit
 #
 side-by-side = true # Display a side-by-side diff view instead of the traditional view
 # navigate = true # Activate diff navigation: use n to jump forwards and N to jump backwards
 relative-paths = true # Output all file paths relative to the current directory
 file-style = yellow
 hunk-header-style = line-number syntax

Code Snippet 2: My configuration for delta in .gitconfig

Gujarati Transliteration

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Mon, 27 Jun 2022 18:43:00 -0400

You can phonetically write a non-English language on an English keyword in Emacs, and that transforms into that non-English script. This is called transliteration, and I demonstrate that for the Gujarati language in this post.

Table of Contents

Enabling transliteration
Toggling the input method
Caveats with Gujarati and other Indic language transliteration
“Input method” cheat sheet
Closing
References

Emacs provides the transliteration feature using the set-input-method command. I’ll introduce that and few related functions in this post to get to help get started with transliteration quickly.

Enabling transliteration #

Emacs uses the “input method” feature to do character conversion from ASCII to the target language or script. The “input method”, stored in current-input-method, is nil by default. In this state, you see the exact ASCII in Emacs buffer, that you typed on the keyboard¹.

In this post, my target non-English language is Gujarati. So I want to type on my English keyboard and have Gujarati script letters show up in the buffer.

Emacs provides the set-input-method command to change the current input method. This command is bound to C-x RET C-\ by default. Pick the new input method after calling that command.

To see the available input methods, do M-x list-input-methods.

As I want to do Gujarati transliteration, I pick the gujarati-itrans method.

If you don’t know Gujarati, don’t fret! The commands shown here will work when transliterating to other languages too — only the Gujarati-specific input method gujarati-itrans will change to the input method of your choice.

Toggling the input method #

I often need to switch between the Gujarati and English languages in the same document. You can see me doing that in this post next section onwards. The toggle-input-method command bound by default to C-\ is helpful here.

So if I am already in the “Gujarati transliteration mode” calling this command will set current-input-method back to nil. Repeating that same call will again set current-input-method to gujarati-itrans, and I will once again be in the “Gujarati transliteration mode”.

Caveats with Gujarati and other Indic language transliteration #

Apologies, but this section is meaningful only if you know how to read Gujarati. So you can safely skip to the next section.

Below table is a quick glimpse of some nuances in Gujarati transliteration. I will save my explanation and instead show some of the mistakes I made in transliteration using examples.

ASCII input	Gujarati Transliteration	Rough Translation
ram	રમ્	(incorrect spelling, no meaning)
rama	રમ	play
raama	રામ	a popular name Raama (as in Lord Raama)
angreji	અન્ગ્રેજિ	(incorrect spelling)
hu.n	હું	I
chhu.n	છું	am
a.ngrejii	અંગ્રેજી²	English (language)
Ime.cksa	ઈમૅક્સ	this literally reads “Emacs”

“Input method” cheat sheet #

Thankfully Emacs provides full help through the describe-input-method command bound to C-h C-\ If you haven’t already noticed the consistency in these bindings, the default bindings with C-\ in them are related to “input method” commands. by default.

For example, M-x describe-input-method gujarati-itrans gives this:

Figure 1: Partial screen capture of Gujarati transliteration cheat sheet C-h C- gujarati-itrans

The ∗Help∗ that shows up looks formidable at the first glance. Though, I found comfort in the fact that roughly half of the key sequences were obvious and roughly half resulted in Gujarati characters that I have never found the need of! 😃

Closing #

Typing this in the “transliteration mode”:

maaru naama kaushala chhe. mane e jaaNii ne aana.nda thaaya chhe ke hu.n aa sahelaaI thI lakhI shaku chhu.n. (joDanI-bhula maapha.)

will result in:

મારુ નામ કૌશલ છે. મને એ જાણી ને આનંદ થાય છે કે હું આ સહેલાઈ થી લખી શકુ છું. (જોડણી-ભુલ માફ.)

Translation: My name is Kaushal. I am happy knowing that I can write this easily.

References #

Emacs Info: Input Methods
- Emacs Info: Select Input Method
M-x describe-input-method gujarati-itrans

I am assuming an English keyboard here. ↩︎
This spelling might still render incorrectly on your browser depending on the unicode character set available for Gujarati on your system. ↩︎

Cleaning up ${GOPATH}/pkg/

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Sat, 25 Jun 2022 10:48:00 -0400

Use go clean -modcache to clean up all the old auto-downloaded Go modules in ${GOPATH}/pkg/. That’s all you need to know. Rest of the post gives just the history of how I got there.

Table of Contents

The $GOPATH directory
The disk space issue
Write-protected ${GOPATH}/pkg/
Cleaning up “modcache”
Wrapping up
References

As I mentioned in the post description above, the solution to clean up the auto-populated ${GOPATH}/pkg/ is to run go clean -modcache. While the solution is simple, it wasn’t easy for me to discover, and so I am writing this short piece to make it a bit more discoverable for others like me.

The `$GOPATH` directory #

The GOPATH environment variable lists places to look for Go code.

If you haven’t set this variable and if the ${HOME}/go/ directory isn’t used to contain the Go distribution, $GOPATH defaults to that path.

When you run go get to install any Go package,

${GOPATH}/pkg/ gets populated with all the Go module dependencies,
.. and all package’s executables if any get installed to ${GOPATH}/bin/.

I think that if the user has set the $GOMODCACHE environment variable, that directory would get populated with Go module downloads instead. But for my use, I am sticking with only the $GOPATH for simplicity.

To confirm the value of GOPATH used by Go, see go env | rg GOPATH.

The disk space issue #

I have been happily installing and building Go apps like Hugo using go get or go build, and all these installations would end up in the $GOPATH. But over time, I noticed that the disk space used by ${GOPATH}/pkg/ just kept on creeping up.

Today I happened to notice that this directory was taking up roughly 4 GB of my disk space! I started analyzing why it was taking up so much space using my favorite tool for this purpose – ncdu.

Here’s a snapshot showing disk usage by one of the sub-directories under ${GOPATH}/pkg/, which shows the problem — Over time, I had accumulated multiple versions of multiple packages!

Figure 1: Snapshot of ncdu showing disk usage for a ${GOPATH}/pkg/ directory

Write-protected `${GOPATH}/pkg/` #

I identified the problem. And I knew that I just needed to delete all the old packages.

But Go wouldn’t allow deleting those pkg/ directories!

I had been searching a solution to this on and off, but didn’t have much success, mainly attributed to the short and generic name of the “Go” language, and the fact that I didn’t know what the ${GOPATH}/pkg/ directory was called.

Cleaning up “modcache” #

Today, I finally had success when I searched for this magic phrase: golang “cannot remove” “pkg/mod” .. and Golang Issues # 27161 was the first search result!

The solution was so simple, but so difficult to look for ..

go clean -modcache

Code Snippet 1: Command to delete contents in ${GOPATH}/pkg/ or the Go modcache

From that issue, I also learned that the ${GOPATH}/pkg/ directory is the default “modcache” or the cache directory for holding all the installed Go modules.

Wrapping up #

Knowing that the stuff I was trying to delete is called modcache, this of course works ..

go help clean | rg modcache -A 2

Code Snippet 2: Getting help with cleaning up the modcache from go help

The issue referenced above gets resolved when a -modcacherw switch gets added to the go build command. I see that switch when I run go help build. But that switch is not available for go get? .. because I don’t see it when I run go help get.

I don’t understand why Go decided to make this so complicated by taking away the write access from the user who installed the Go package!

At least go clean -modcache accomplishes what I want .. sigh

References #

View GitHub Pull Requests in Magit

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Thu, 23 Jun 2022 17:51:00 -0400

How to view GitHub Pull Request branches locally in the cloned repo, and more importantly, how to do that automatically from within Emacs.

Table of Contents

Locally creating a branch for a PR
Getting references to all Pull Requests
Automatically adding PR refs
Summary
References

I have a few public projects in git repos, but I don’t get that much traffic in Pull Requests (PR) Gitlab calls these Merge Requests or MRs. . So when I need to add additional commits to a PR, I would just add the PR author’s remote to my local repo, check out their PR branch, add my own commits and then merge that to my project’s main branch.

As these occurrences were few and far apart, I didn’t have a need to view the PR branches directly from within Emacs/Magit. Though, I somehow knew that each GitHub Pull Request’s HEAD got assigned a git reference. But I didn’t need to use that knowledge until today 😃.

Today, when discussing PR # 20 on Prot’s Denote package’s GitHub mirror Prot uses SourceHut as the primary git forge for his Emacs packages. But I am glad that he doesn’t mind the activity in Issues and Pull Requests on the GitHub mirror. , he wrote this comment:

Now I just need to figure out how best to incorporate your changes into the org-id branch so I can add the final bits. I am not too familiar with the PR workflow …

.. and that inspired this post today.

Locally creating a branch for a PR #

From the GitHub docs, the git command to create a local branch for a PR is this:

git fetch origin pull/ID/head:BRANCHNAME

I have the denote package cloned from its GitHub mirror. So the origin remote’s url is https://github.com/protesilaos/denote.

Make sure that the remote name used in this command is pointing to a GitHub repo, and not a mirror forge like GitLab or SourceHut.

When I ran the below command, I got a new branch pr-20 pointing to the latest commit of that PR:

git fetch origin pull/20/head:pr-20

Awesome!

.. But that wasn’t good enough
.. Now I wanted more
.. I didn’t want to manually create a branch for each PR.

Getting references to all Pull Requests #

Now that I was on that quest of “I want more”, it didn’t take me long to re-discover this 7-year old nugget by Oleh Krehel. Here are the relevant bits from that post:

Open the local repo’s .git/config file.
Find the [remote "origin"] section

Modify it by adding this one line with pull refs. This is the same for all GitHub repositories.

[remote "origin"]
 url = https://github.com/USER/REPO.git
 fetch = +refs/heads/*:refs/remotes/origin/*
 fetch = +refs/pull/*/head:refs/pull/origin/*

With that edit in place, when I did l a (show the log for all git references), followed by f a (fetch all the remotes) in the Magit, I could see the references to the denote repo’s PRs!

Figure 1: Viewing PR references from denote package’s GitHub repo

.. But that still wasn’t good enough
.. I didn’t want to manually edit the .git/config in each repo.

Automatically adding PR refs #

Of course, I wasn’t the first one to think of this!

Another Emacs veteran Artur Malabarba had already had this covered also around 7 years back. Coincidentally, that post was written as a response to that same blog post by Oleh where he shared the above .git/config tip.

In that post, Artur shares an Emacs Lisp function that uses Magit functions like magit-get, magit-get-all and magit-git-string to auto-add the fetch = +refs/pull/*/head:refs/pull/origin/* line in the .git/config. This magic happens after checking that the origin remote points to a GitHub repo, and if that line doesn’t already exist.

Here, I am lightly modifying the function shared in that post so that the origin remote name is not hard-coded The reason is that sometimes, I name the original remote as upstream and my fork as fork, and I might have no remote named origin. . Credit for the main logic in this code still goes to Artur.

(defun modi/add-PR-fetch-ref (&optional remote-name)
 "If refs/pull is not defined on a GH repo, define it.

If REMOTE-NAME is not specified, it defaults to the `remote' set
for the \"main\" or \"master\" branch."
 (let* ((remote-name (or remote-name
 (magit-get "branch" "main" "remote")
 (magit-get "branch" "master" "remote")))
 (remote-url (magit-get "remote" remote-name "url"))
 (fetch-refs (and (stringp remote-url)
 (string-match "github" remote-url)
 (magit-get-all "remote" remote-name "fetch")))
 ;; https://oremacs.com/2015/03/11/git-tricks/
 (fetch-address (format "+refs/pull/*/head:refs/pull/%s/*" remote-name)))
 (when fetch-refs
 (unless (member fetch-address fetch-refs)
 (magit-git-string "config"
 "--add"
 (format "remote.%s.fetch" remote-name)
 fetch-address)))))
(add-hook 'magit-mode-hook #'modi/add-PR-fetch-ref)

Code Snippet 1: Function to auto-add GitHub PR references to the repo's .git/config

Summary #

With the above snippet added to your Emacs config and evaluated, each time you visit a repo cloned from GitHub in the Magit Status buffer (M-x magit-status), the PR refs will get auto-added to that repo’s .git/config if needed.

After that, you can easily view the commits from all the PRs by doing l a f a.

References #

Gujarati fonts in Emacs

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Sun, 19 Jun 2022 01:31:00 -0400

Setting a different font for a specific script or language in Emacs.

Table of Contents

Setting a “fontset” font
Gujarati fonts

All Emacs versions ship with a nifty HELLO file that you can quickly open using M-x view-hello-file or its default binding C-h h. This file lists “Hello” written in dozens of languages to demonstrate some of the character sets supported by Emacs.

Figure 1: “Hello” buffer in Emacs

Born and raised in the Gujarat state in India, I grew up speaking the Gujarati (ગુજરાતી) language India officially recognizes 22 languages (as of <2022-06-19 Sun>) and Gujarati is one of them. and so it’s also the language closest to my heart. So I was pleasantly surprised to see a representation of Gujarati in the “Hello” buffer! In the above screenshot, in the “South Asia” section, the script after the yellow cursor is Gujarati, and it reads namaste.

Setting a “fontset” font #

It was on <2018-08-13 Mon> that I discovered the presence of Gujarati script in that “Hello” buffer, and the reason I know that exact date is because I had asked a question regarding that on the help-gnu-emacs mailing list 😃.

This was the time when Emacs was using the m17n library for multi-lingual font rendering by default. The question was regarding a font rendering issue I was seeing. As I learn later in that thread, it was because I didn’t have the m17n database installed on my machine At least in 2022, the harfbuzz library is the recommended library for text shaping and font rendering. Someone please correct me if that’s wrong. In any case, I have switched to using harfbuzz instead of m17n for a while now and haven’t found any font-rendering issues with non-English scripts. . But it’s in that support thread that, thanks to Andy Moreton, I learned that you can change the font for the Gujarati script using set-fontset-font.

This applies in general to any script. You can read more details about this function in Emacs Info: Modifying Fontsets, but here’s the gist:

(set-fontset-font "fontset-default" 'gujarati "<FONT NAME>")

Code Snippet 1: Setting default font for Gujarati script using set-fontset-font

That led me down the path of exploring the available Gujarati fonts out there ..

Gujarati fonts #

After looking around for a bit, I found a wonderful collection of Gujarati fonts in this GitHub repository: github.com/samyakbhuta/chhapkaam The repo name is chhapkaam (I would have spelled it as chhaapkaam) which is the Gujarati word છાપકામ, meaning “printing”. .

Below is my further curated list of fonts from the above list:

Table 1: Gujarati Fonts

Font Name	Category	Homepage	Download
Shruti	serif		wfonts.com
Mukta Vaani	serif	Ek Type – Mukta Vaani	GitHub
Lohit Gujarati	sans serif	Pagure – Lohit	pagure releases

To use these fonts, after downloading and installing them on your system, evaluate Code Snippet 1 above with the correct “FONT NAME”. For example, to set the Gujarati text to use the Shruti font, evaluate (set-fontset-font "fontset-default" 'gujarati "Shruti").

Thanks for reading (વાંચવા બદલ આભાર) 🙏!

Emacs Lisp: Advice Combinators

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Fri, 17 Jun 2022 21:30:00 -0400

My diagrammatic take on summarizing all the Emacs advice combinators.

Table of Contents

Overview on using advices
1 :before
2 :after
3 :override
4 :around
5 :before-while
6 :before-until
7 :after-while
8 :after-until
9 :filter-args
10 :filter-return
Summary
References

If you have read some of my earlier posts, you would know that I really enjoy using the Emacs Advice system 😃.

The “advice” feature lets you add to the existing definition of a function, by advising the function. This is a cleaner method than redefining the whole function, because it’s easier to debug and if you don’t need it, you can just remove the advice.

You can jump to the References section below if you need to look at the related sections in the Emacs Lisp Manual.

Overview on using advices #

I do not plan to write a tutorial on how to write advices in Emacs-Lisp, but here’s a 3-second primer:

To add an advice: (advice-add 'original-fn <combinator> #'advising-fn)
To remove an advice: (advice-remove 'original-fn #'advising-fn)

This article attempts to briefly describe different ways of advising a function, using 10 different combinators. If you have never used advices in your Emacs config, don’t worry. I am hopeful that the diagrams in this post and the examples linked for some of the combinators in the Summary section makes this concept a bit easier to assimilate.

Diagram Legend

Initial black circle: Original Fn input arguments
Yellow box: Original Fn
Gray box: Advising Fn
Black circle inside a white circle: Return values

1 `:before` #

Figure 1: :before advice

2 `:after` #

Figure 2: :after advice

3 `:override` #

Figure 3: :override advice

4 `:around` #

Figure 4: :around advice

5 `:before-while` #

Figure 5: :before-while advice

6 `:before-until` #

Figure 6: :before-until advice

7 `:after-while` #

Figure 7: :after-while advice

8 `:after-until` #

Figure 8: :after-until advice

9 `:filter-args` #

Figure 9: :filter-args advice

10 `:filter-return` #

Figure 10: :filter-return advice

Summary #

Here’s a concise summary of what each advice combinator does. For brevity, the advising function is called A and the original function is called O.

Once you click on any of the example posts, search for advice-add on that page to find the code example.

Table 1: Summary of what each advice combinator means

Combinator	Description	Example
`:before`	A is called before O. O args and return values are not modified.
`:after`	A is called after O. O args and return values are not modified.
`:override`	A is called in lieu of O. A gets the same args as O.	Zero HTML Validation Errors!
`:around`	A is called in lieu of O. A gets O fn + O args as args.	Using Emacs advice to silence messages from functions
`:before-while`	A is called first. If it returns non-nil, O is called.
`:before-until`	A is called first. If it returns nil, O is called.	Org: Show only Post subtree headings
`:after-while`	O is called first. If it returns non-nil, A is called.
`:after-until`	O is called first. If it returns nil, A is called.
`:filter-args`	A is called first. O is called next with return value from A as input.	Narrowing the Author column in Magit
`:filter-return`	O is called first. A is called next with return value from O as input.	Zero HTML Validation Errors!

If you have any feedback on how these diagrams can be made easier to understand, please let me know.

References #

Elisp Info: Advising Functions
- Elisp Info: Advice Combinators

Org: Show only Post subtree headings

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Thu, 16 Jun 2022 00:21:00 -0400

How to define a custom org-global-cycle-like command that collapses only the Org subtrees with specific properties.

Table of Contents

Org Global Cycle
Skeleton of only Post headings
The “Collapse All Posts” function
Binding with C-u C-c TAB
Result

I start this post by introducing what the Org mode global cycling command does, what kind of subtree folding I actually need, and then share the solution with code snippets.

Org Global Cycle #

Org mode has a built-in org-global-cycle command that you might be familiar with. It’s bound by default to the S-TAB key. Each time this command is called, the Org buffer will cycle through these states:

Overview: Show only the Level 1 headings and collapse everything underneath.
Contents: Show only the Org headings and collapse all the content.
Show All: Expand all the headings and show their contents too.

If a numeric prefix N is used with this command, it will show only the Org headings up to Level N. For example, C-2 S-TAB will show only the headings up to Level 2.

This is a really helpful command, but I needed something different ..

Skeleton of only Post headings #

I maintain most of this website’s content in a single Org file. I have dozens of blog posts organized in Org subtrees, which I further organize under “category” headings .. It kind of looks like the below mock-up: It’s amazing how many features PlantUML has. If you are interested in creating diagrams like these, check out the PlantUML Salt syntax.

Figure 1: Post Subtrees at arbitrary heading levels

As we can see,

All the post subtrees are not at Level 1 headings.
They are also not at a fixed Level N.
The heading level of the post depends on how many parent categories that post has (and that will also change over time).

I needed to basically show everything leading up to a post subtree heading, and then collapse all the content under that post; even the sub-headings.

The “Collapse All Posts” function #

The modi/org-hugo-collapse-all-posts function defined below meets the above requirement:

It first widens the whole buffer and expands all the headings.
Then it loops through all the headings and collapses all the post subtrees i.e. all the subtrees that have the EXPORT_FILE_NAME property set. This is where I use the org-map-entries magic.
Finally it looks for Org headings that begin with “Footnotes” or “COMMENT” and collapses them as well.

I am using the development version of Org mode (version 9.6, yet to be released as of <2022-06-15 Wed>) which has the new org-fold library. This library obsoletes the use of outline.el library and other code-folding related functions in Org mode. So cl-flet is used to create function symbol aliases that use the org-fold-* functions if available, otherwise they fall back to the legacy functions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


(defun modi/org-hugo-collapse-all-posts ()
 "Collapse all post subtrees in the current buffer.
Also collapse the Footnotes subtree and COMMENT subtrees if
present.

A post subtree is one that has the EXPORT_FILE_NAME property
set."
 (interactive)
 (cl-flet ((show-all (if (fboundp 'org-fold-show-all)
 #'org-fold-show-all
 #'org-show-all))
 (hide-subtree (if (fboundp 'org-fold-hide-subtree)
 #'org-fold-hide-subtree
 #'outline-hide-subtree)))
 (widen)
 (show-all '(headings))
 ;; Collapse all the post subtrees (ones with EXPORT_FILE_NAME
 ;; property set).
 (org-map-entries #'hide-subtree "EXPORT_FILE_NAME<>\"\"" 'file)
 ;; Also hide Footnotes and comments.
 (save-excursion
 (goto-char (point-min))
 (while (re-search-forward "^\\(\\* Footnotes\\|\\*+ COMMENT\\)"
 nil :noerror)
 (hide-subtree)))))

Code Snippet 1: Function that collapses all the post subtrees in the current buffer

Binding with `C-u C-c TAB` #

The function is ready, but let’s now add a bit of convenience to it.

If a point is under a subtree, C-c TAB will collapse that subtree while showing only Level 1 headings, and if a numeric prefix is used, it will show only those many levels of headings. I decided to bind the above function to C-u C-c TAB because,

The behavior of modi/org-hugo-collapse-all-posts falls in the same category as that of C-c TAB.
The C-u C-c .. binding rolls off the fingers pretty nicely 😃.

This binding is achieved using one of my favorite Emacs features .. the advice system. The :before-until Advice Combinator is used here, which means that if the advising function (below) returns a nil, the advised or the original function org-ctrl-c-tab is not called.

The advising function below detects if the C-u prefix argument is used. If it is, the modi/org-hugo-collapse-all-posts function is called, otherwise the original org-ctrl-c-tab function is called.

1
2
3
4
5
6
7
8
9


(defun modi/org-ctrl-c-tab-advice (&rest args)
 "Run `modi/org-hugo-collapse-all-posts' when
doing \\[universal-argument] \\[org-ctrl-c-tab]."
 (let ((do-not-run-orig-fn (equal '(4) current-prefix-arg)))
 (when do-not-run-orig-fn
 (modi/org-hugo-collapse-all-posts))
 do-not-run-orig-fn))

(advice-add 'org-ctrl-c-tab :before-until #'modi/org-ctrl-c-tab-advice)

Code Snippet 2: Bind C-u C-c TAB to call modi/org-hugo-collapse-all-posts

Result #

After evaluating the above two snippets, when I do C-u C-c TAB in my “blog posts” Org buffer, I see this:

Figure 2: My “blog posts” Org buffer showing only the Post subtree headings

It matches that earlier mockup — Mission accomplished! 💯

Firefox: Always open a New Tab after Current

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Wed, 15 Jun 2022 02:08:00 -0400

The default behavior of Firefox opening new tabs from links had been troubling me for a while .. it was time I tried to do something about it.

Table of Contents

The Issue
Expected Behavior
False Leads
Firefox Bug # 1485683
Solution

This was one of the those little things in software that bug me, but I don’t start looking for a solution right-away because of reasons like “I am too busy with something else”, or “The bug is not so bad.. let me see if I can live with it”.

But no .. this particular issue with how Firefox dealt with opening new tabs from links was now getting on my nerves.

The Issue #

While visiting a page, I typically middle-click on links so that they open in new tabs .. and I do that a lot! And this is the order of where the new tabs would open:

Figure 1: Unexpected order of opening of new tabs opened after each middle-click

In my view, this behavior is completely wrong!

As you can see above, if I already have a bunch of links open (see the last row where “Tab 1”, “Tab 2” and “Tab 3” are already open), the last opened tab will land up between those tabs and a sea of other previously opened tabs! When you have over a dozen of tabs already open, hunting for the last opened tabs is really annoying.

Expected Behavior #

I would instead expect for each middle-click to open the new tab immediately to the right of the current tab. Below diagram shows it very clearly that this behavior would save me the exercise of hunting for that new tab.

Figure 2: Expected order of opening of new tabs opened after each middle-click

False Leads #

So I searched online for “firefox new tab location” and similar search terms, and found Q&A on Mozilla Support like How do I make tabs open on the right by default? and New tabs ALWAYS open to the right of the last open tab, not to the right of current tab.

The “Chosen solutions” for both of these issues suggested setting browser.tabs.insertAfterCurrent = true and browser.tabs.insertRelatedAfterCurrent = true (which is the default) in the Firefox about:config.

But that didn’t work!!

Setting browser.tabs.insertAfterCurrent = true affected the “new tab opening” behavior when opening only blank new tabs using Ctrl+T or hitting the “New Tab” button. This setting had nothing to do with the location of new tabs opened by the “Open Link in New Tab” or “middle-click on link” action.

Firefox Bug # 1485683 #

After scouring through many search results, I landed up on Firefox Bug # 1485683: browser.tab.insertaftercurrent order which reported the exact same issue I was seeing 🎉.

So I replied to that bug thread confirming that I still saw that issue on Firefox 101.0.1 .. 4 years later .. just in case that motivates someone to still fix it.

But my issue isn’t fixed yet.

Solution #

Finally, the Open Tabs Next to Current (repo) Firefox Add-on saved the day!

The introduction of this add-on incorrectly states this:

After the addition of the browser.tabs.insertAfterCurrent setting in about:config you do not need this extension anymore.

But I still tried installing it to see .. just in case .. 🤞 ..

And it worked!!

I could finally get new tabs to open in the order as shown in the Expected Tab Order figure above 😂.

Thank you Sebastian Blask for Open Tabs Next to Current!

Saving Python pip dependencies

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Tue, 14 Jun 2022 00:38:00 -0400

How I generated the Python dependencies file requirements.txt so that Netlify can install and run the HTML5 Validator before deploying this site.

Table of Contents

1 Create a virtual environment
2 Activate the virtual environment
3 Install the packages using pip
4 Generate requirements.txt
Netlify deployment
Summary

Recently I learned about the Python tool html5validator tool and used it to run HTML5 validation on local HTML files. You can read more about that in the Offline HTML5 Validator post.

But then I wondered .. “Wouldn’t it be awesome if I run this validation step each time after Hugo generates this site on Netlify, but before it gets deployed?”. That curiosity led me to Netlify’s documentation on Python dependencies, and I learned that Netlify will run pip install to install the dependencies in requirements.txt present in the repository’s base directory.

I followed the pip freeze > requirements.txt step in that documentation but that ended up listing all my pip installed packages in that file! I needed the requirements.txt to include the dependencies only for html5validator. This post was born in my quest to achieve that.

The solution to this problem was to first create a Python virtual environment in my site directory, and then do all the pip operations in there. I learned about this virtual environment step from this Python Pandemonium post.

The venv documentation has a lot of details — I’ll just list the key steps in this post.

1 Create a virtual environment #

cd to your site directory and run the below command create a virtualenv directory named pyenv in there.

python3 -m venv pyenv

As we are creating this virtual environment just for the sake of creating a requirements.txt, we don’t need to commit this directory to git.

2 Activate the virtual environment #

The virtualenv directory pyenv/bin/ will have multiple flavors of shell scripts to activate your virtualenv. I am using activate.csh here as my shell is tcsh 🙄.

source pyenv/bin/activate.csh

Just to emphasize, it is important to use the source command here, and not just call the shell script directly.

Once you activate this virtualenv, you should see something like [pyenv] in the shell prompt.

3 Install the packages using `pip` #

Install all the project-specific packages. In this case it was just this:

pip install html5validator

The package (and its dependencies) will be installed in pyenv/lib/python3.7/site-packages/. Here, the python3.7/ sub-directory name will match the version of Python you have installed.

4 Generate `requirements.txt` #

Finally, we run the pip freeze command mentioned in Netlify docs, but with the --local switch:

pip freeze --local > requirements.txt

The --local option ensures that globally-installed packages are not listed even if the virtualenv has global access.

Here’s the requirements.txt created by that command:

html5validator==0.4.2
PyYAML==6.0

Code Snippet 1: The requirements.txt for Netlify

Now, I could have just manually typed html5validator==0.4.2 in a requirements.txt and committed that, and that would work too. But then, I wouldn’t have learned how to create a project-specific pip dependency file 😄.

Netlify deployment #

The html5validator has a dependency on Java 8. Luckily the Netlify environment already comes with that installed. So the only extra setup needed to make this package work on Netlify was to set the PYTHON_VERSION environment variable to 3.8 The version number should be one of the values listed in Netlify’s included software list. .

Summary #

The numbered headings in this post already summarize the steps needed to create a project-specific requirements.txt.

With these in place:

✅ requirements.txt committed in site directory root
✅ Netlify environment variable PYTHON_VERSION set to 3.8

my Netlify deployment script now does HTML5 Validation along with few other checks before this site gets deployed 🎉.

Here’s the full build.sh script.

Disarming the 'tar' bomb in 10 seconds

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Mon, 13 Jun 2022 01:14:00 -0400

Use tar -caf <file> <dir> to create an archive, tar -xf <file> to extract one, and more.

Table of Contents

Mnemonics
Creating an archive
Extracting an archive
Listing contents of an archive
Summary

I had come across this post by user Garrit on Mastodon, and that inspired this post.

In reply to:

https://garrit.xyz/posts/2022-06-02-tar-commands

A post on the Unix command tar cannot leave out the obligatory xkcd tar comic 😄, so here it is:

xkcd.com

Mnemonics #

These few mnemonics help me remember the basic and my most frequently used tar options:

c to (c)reate
a to (a)uto compress the archive based on the file name
x to e(x)tract
f for the archive (f)ile name we are dealing with (whether creating, listing or extracting an archive)

.. and a few not so frequent options (for me):

t to lis(t) contents of an archive
v for (v)erbose output

Creating an archive #

tar -caf <file> <dir>

A keen user might have noticed that I am using tar -caf .. instead of tar caf .. i.e. I am using a hyphen before the tar options. Both approaches work and they look similar, but the approach with the hyphen is the newer Short Option style while the other is the Old Option style.

I prefer the short option style because .. well.. the other style is old.. and also because the short option style is stricter e.g. the -f switch has to be followed by the file name.

Whether you are creating a regular .tar archive, or a compressed archive like .tar.gz, always use the auto-compresion switch -a. That relieves you from deciding if you need that switch, or which compression algorithm switch should be used 😄.

The -a switch makes the decision for you based on the file extension. For example, tar -caf foo.tar foo/ will create a regular archive, while tar -caf foo.tar.gz foo/ will create a compressed archive using gzip. You can read more about it in Creating and Reading Compressed Archives.

Extracting an archive #

tar -xf <file>

At times, it might be useful to add the verbosity switch -v to this command and do tar -xvf <file>.

Just to reiterate, the -f switch must be followed by the archive file name.

Listing contents of an archive #

tar -tf <file>

Once you are done creating an archive, you might feel the need to check if the archive contains everything you expect. Or you might want to check what’s inside the archive before you extract it.

This command is often paired with grep or rg (ripgrep) like so: tar -tf foo.tar.xz | rg 'some_file_name_in_archive'.

Summary #

If you simply glossed over the whole article, or didn’t read through the all linked manual pages I know you didn’t 😉 , just remember this —

tar -caf to create and tar -xf to extract

Unclutter: A better Reader View for browsers

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Sun, 12 Jun 2022 23:07:00 -0400

Unclutter is a better looking Reader View browser add-on available for both Firefox and Chrome.

Table of Contents

Why?
Caveat
Closing

I am writing this post to share this wonderful browser add-on Unclutter (available for both Firefox and Chrome) by Peter Hagen. I discovered this a few days back on Hacker News. But given how awesome this add-on is, I am surprised that the HN post didn’t gain any traction.

I’ll quickly post the important links related to this add-on, and then briefly list few points on why I like it.

👉 Unclutter Homepage | Source code | Add-ons: Firefox, Chrome 👈

Why? #

I use Firefox as my primary browser. If you are too, you might be wondering “why install this add-on when Firefox already has a Reader View”.

Here are my reasons:

You do not lose the article’s origin style!
The Unclutter author Peter Hagen was super-responsive when I opened an issue on his repo. This add-on did not work well on one of the pages on this site, but he fixed it in less than 24 hours!
This website shows an outline on the side, but many websites don’t. Unclutter adds a similar outline for any page where it’s enabled.
Unclutter can be auto-enabled for your chosen sites. For example, ever since I have discovered this add-on, I have auto-enabled it on https://www.masteringemacs.org.
You can customize a key-binding or shortcut to toggle this add-on. For now, I am keeping the default binding Alt + C.
The subtle animation you see when enabling/disabling this add-on is pretty cool.
Like the Firefox built-in Reader View, Unclutter also removes distractions – So no ads or pop-ups!

It has other features like annotating pages with your notes privately and displaying comments from Hacker News and Hypothes.is, but I don’t use those.

Caveat #

While viewing a page with Unclutter enabled, there are no issues. But I have noticed that if I toggle it off, the original CSS gets broken. This issue is tracked on its repo. Until this gets fixed, a quick workaround is to do Ctrl + F5 (force reload the page and thus its CSS too) after disabling Unclutter.

Closing #

But this minor issue doesn’t prevent it from making it my default “reader view” app on Firefox.

I’ll end this post with a before and after when visiting this page. Feel free to click those images to view them in higher resolution.

Zero HTML Validation Errors!

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Sun, 05 Jun 2022 17:58:00 -0400

How I fixed my site content and went down from 46 HTML validations errors down to 0!

Table of Contents

1 Avoid duplicate heading id attributes
2 Remove inline <style> elements
3 Ensure that all images have captions or alt attributes
4 Do not have hyperlinks in headings
Validation Ignores
- 5 Ignore errors due to hyperlinks in inline SVG
- 6 Ignore files not expected to serve HTML content
Summary

In my previous HTML5 Validator post, I mentioned:

I was a bit disappointed to see validation errors on my site, but then it wasn’t too bad .. 46 errors.

But they truly say ..

Ignorance is bliss.

So .. once I realized that my site had 46 validation errors, I lost that bliss .. and I couldn’t rest easy — I had to fix them all 😁.

This post summarizes the categories of those errors and how I fixed The fixes mentioned in this post refer to changes in Org mode content. But you should be able to derive equivalent fixes for Markdown too. them all.

1 Avoid duplicate heading `id` attributes #

"file:/public/notes/string-fns-nim-vs-python/index.html":636.34-636.46: error: Duplicate ID "notes".
"file:/public/notes/nim-fmt/index.html":300.34-300.52: error: Duplicate ID "older-issue".
"file:/public/notes/nim-fmt/index.html":306.20-306.33: error: Duplicate ID "floats".
"file:/public/page/6/index.html":29.39-29.54: error: Duplicate ID "fnref:1".

Errors with above kind of signatures were fixed by,

Converting headings to description lists

I had a bunch of generic headings like “Notes” and “Older Issue” in some of my posts. After taking a second look at those, it made more sense to convert those to description lists. So in Org mode, I converted headings like * Notes to description lists - Notes ::.
Setting CUSTOM_ID heading property

For the cases, where the headings needed to be left as so, their IDs were uniquified by setting their CUSTOM_ID property. For example, below fixed the Duplicate ID “floats” errors.
```
 ** Precision
 ..
 *** Floats
+:PROPERTIES:
+:CUSTOM_ID: precision-floats
+:END:
 ..
 ** Type (only for numbers)
 ..
 *** Floats
+:PROPERTIES:
+:CUSTOM_ID: type-floats
+:END:
```
Code Snippet 1: Using CUSTOM_ID property to uniquify heading ID's
Prevent footnote links in post summaries

This issue was due to me not being conscious about how the footnote references work in a post versus on a page outside that post’s context. The issue was caused by footnote references getting into the post summaries parsed by Hugo, which will then show up on the list pages.

The fix was simple — Edit the post summaries so that they don’t contain any footnote references.

2 Remove inline `<style>` elements #

"file:/public/grep-po/index.html":51.139-51.145: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/how-do-i-write-org-mode/index.html":23.194-23.200: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)

Errors with above kind of signatures were fixed by,

Avoiding export of raw <style> elements in the Markdown content

I figured out which functions were responsible for injecting <style> elements in Markdown content and then advised them to stop that. After applying these advises, I lost the in-content rules for CSS classes .org-center and .csl-entry. So I put those rules directly in this website’s CSS.

(defun modi/org-blackfriday-center-block (_center-block contents info)
 (let* ((class "org-center"))
 (format "<div class=\"%s\">%s\n\n%s\n</div>"
 class (org-blackfriday--extra-div-hack info) contents)))
(advice-add 'org-blackfriday-center-block :override #'modi/org-blackfriday-center-block)

(defun modi/org-cite-csl-render-bibliography (bib-str)
 (replace-regexp-in-string "<style>\\.csl-entry[^<]+</style>" "" bib-str))
(advice-add 'org-cite-csl-render-bibliography :filter-return #'modi/org-cite-csl-render-bibliography)

Code Snippet 2: Emacs-Lisp advices to prevent <style> elements in exports

Removing unnecessary micro-styling

I found a single case, where an inline CSS rule was defined in content for CSS class .repr-type for a table. I just removed that without affecting the looks of that rendered table too much.

3 Ensure that all images have captions or `alt` attributes #

"file:/public/hugo-use-goat-code-blocks-for-ascii-diagrams/index.html":24.130-24.255: error: An "img" element must have an "alt" attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.

Errors with above kind of signatures were easily fixed by ensuring that all images had captions Thankfully, there were only two images that were missing captions. . The Hugo figure shortcode adds the caption to the alt attribute if the alt is not specified separately.

As an example, here’s how I fixed the above error:

+ #+name: fig__disproportionate_box_drawing
+ #+caption: Disproportionate box drawing characters
[[file:images/hugo-use-goat-code-blocks-for-ascii-diagrams/ascii-diagram-rendered-in-plain-text-code-block.png]]

Code Snippet 3: A git diff showing addition of caption to an image

4 Do not have hyperlinks in headings #

"file:/public/using-emacs-advice-to-silence-messages-from-functions/index.html":151.687-151.732: error: Start tag "a" seen but an element of the same type was already open.
"file:/public/using-emacs-advice-to-silence-messages-from-functions/index.html":151.748-151.751: error: Stray end tag "a".
"file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: Start tag "a" seen but an element of the same type was already open.
"file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: End tag "a" violates nesting rules.
"file:/public/auto-count-100daystooffload-posts/index.html":114.526-114.529: error: Stray end tag "a".

While the hyperlinks in headings work well, they created invalid HTML in the Hugo-generated TOC. So while these errors were created technically because of a bug in Hugo It’s really cool when you end up finding a bug in an upstream project while trying to fix the errors in your own thing 😎. , I wanted to fix these errors on my end as soon as I can.

I reviewed the errors, and this is all it took to get rid of them all:

Remove manually inserting hyperlinks in headings

 show two methods of finding sources of any printed messages.
-***** Using plain-old /grep/ or [[https://github.com/BurntSushi/ripgrep][/rg/]]
+***** Using plain-old /grep/ or /ripgrep (rg)/
 This method is pretty easy (but not robust) to use if the search
 ..
 Org source directory and search for the /"org-babel-exp process .."/
-string ..
+string using [[https://github.com/BurntSushi/ripgrep][~rg~]] ..

Code Snippet 4: Removing hyperlink from a heading

Remove Org Radio links that created links in headings

Here, a Org heading happened to contain the string “Day count”, which was also an an Org Radio link in that post. While ideally that shouldn’t have mattered, I removed that radio link to get around this Hugo bug.
```
 .. /Just may be/. But regardless, I am already enjoying writing once
-again, and it's great to see the <<<Day count>>> (counting up to 100)
+again, and it's great to see the Day count (counting up to 100)
 increase with each new post!
```
Code Snippet 5: Removing an Org Radio link

Validation Ignores #

Above fixes fixed 43 out of 46 errors, but the remaining 3 were unfixable.

5 Ignore errors due to hyperlinks in inline SVG #

"file:/public/notes/plantuml/index.html":114.474-114.678: error: Attribute "title" not allowed on element "a" at this point.

This error was caused by hyperlinks in inline SVG elements. These SVG elements are created by PlantUML. The hyperlinks in SVG feature works great, and as these are generated by PlantUML, I chose to just ignore these errors.

I ignored this error by adding the --ignore-re 'notes/plantuml.*Attribute.*title.*not allowed' switch to the html5validator command.

6 Ignore files not expected to serve HTML content #

"file:/public/googleFOO.html":1.1-1.52: error: Non-space characters found without seeing a doctype first. Expected "<!DOCTYPE html>".

The googleFOO.html file here is not a valid HTML file. It’s a just a cookie file that was used by Google to verify that I own this domain.

This error was masked by adding the --ignore 'googleFOO' switch to the html5validator command.

Summary #

Once I fixed the 43 errors by tweaking the Org mode content, and added those two ignores, I had zero validation errors! 🎉

If you are interested in the fix details, here are the commits.

Offline HTML5 Validator

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Wed, 01 Jun 2022 00:11:00 -0400

Validate your website offline — It’s just one curl command away.

Table of Contents

Using the .jar
- Using pip install
Using pre-compiled binary
Results
Conclusion

I have been using the online HTML5 Validator https://html5.validator.nu/ for few years now. I have a link at the bottom of each post to validate that page’s HTML. For an example, see the html5 validator link at the bottom of this post.

But it didn’t occur to me until now to look for a way to do the same validation offline! Offline validation would be useful so that I can look at any HTML generation problem before I deploy the website. So I looked for a solution online, and of course it’s answered on StackOverflow 😄.

It turns out that same Nu HTML5 Validator project that provides the online validation service, also provides a Java application as well as pre-compiled binaries for Linux, Windows and MacOS for offline use!

To use the Java .jar file, you need to have at least Java 8 installed on your system. But you don’t need to have any version of Java installed if you use the pre-compiled binary instead. See its documentation for more details.

Using the `.jar` #

Requires at least Java 8

Download

Download the latest vnu.jar from the project’s GitHub Releases section.

curl -ROLs https://github.com/validator/validator/releases/download/latest/vnu.jar

Run

Below command runs the validator only on the HTML files in the public/ If you are using Hugo, the hugo command will publish the HTML files in the public/ directory by default. directory. See its Usage documentation for more details.

java -jar vnu.jar --skip-non-html --errors-only public/

For my usecase, if I don’t provide the --skip-non-html --errors-only switches, the output is too noisy.

Using `pip install` #

If you do not want to manually download the .jar, there’s a Python wrapper available to do the same for you: html5validator.

Install

 pip install --user html5validator

Run

 html5validator --root public/

It seems like this Python wrapper implicitly passes --skip-non-html --errors-only to the Java app. So those are not needed when running html5validator. But on the flip side, it needs the --root switch when specifying the directory to run the script on.

Note that you still need to have at least Java 8 installed when running this Python app too, because it downloads and run the same .jar behind the scenes.

Using pre-compiled binary #

If your system doesn’t have the required Java version, you can use the pre-compiled binary instead.

Download & Extract

Download and extract the vnu.<OS>.zip for your OS from the same Releases section. Here, I am showing how to do that on Linux:

curl -ROLs https://github.com/validator/validator/releases/download/latest/vnu.linux.zip
unzip vnu.linux.zip

The extracted binary path will be vnu-runtime-image/bin/vnu.

Run

The run options will be the exact same; just that you will be running the binary directly instead of running through java.

vnu-runtime-image/bin/vnu --skip-non-html --errors-only public/

Results #

<2022-06-05>: This website now has zero validation errors! 🎉 All the errors listed in the collapsed log below are now resolved. See my Zero HTML Validation Errors! post on how I did that.

I was a bit disappointed to see validation errors on my site, but then it wasn’t too bad .. 52 46 errors:

Some I already fixed

These 6 errors were fixed in this commit.

"file:/public/getting-started-with-texlive/index.html":6.2198-6.2206: error: Element "package" not allowed as child of element "li" in this context. (Suppressing further errors from this subtree.)
"file:/public/getting-started-with-texlive/index.html":6.2256-6.2259: error: End tag "li" implied, but there were open elements.
"file:/public/getting-started-with-texlive/index.html":6.2198-6.2206: error: Unclosed element "package".
"file:/public/getting-started-with-texlive/index.html":6.2307-6.2315: error: Element "package" not allowed as child of element "li" in this context. (Suppressing further errors from this subtree.)
"file:/public/getting-started-with-texlive/index.html":6.2316-6.2319: error: End tag "li" implied, but there were open elements.
"file:/public/getting-started-with-texlive/index.html":6.2307-6.2315: error: Unclosed element "package".

Some I can probably fix

 "file:/public/notes/nim/index.html":472.480-472.486: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
 "file:/public/notes/nim/index.html":1359.221-1359.231: error: Duplicate ID "log".
 ..

Some need to be ignored

 "file:/public/google4a938eaf9bbacbcd.html":1.1-1.52: error: Non-space characters found without seeing a doctype first. Expected "<!DOCTYPE html>".
 "file:/public/google4a938eaf9bbacbcd.html":1.1-1.52: error: Element "head" is missing a required instance of child element "title".
 ..

And the rest would be out of my scope to fix

 "file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: Start tag "a" seen but an element of the same type was already open.
 "file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: End tag "a" violates nesting rules.
 "file:/public/auto-count-100daystooffload-posts/index.html":114.526-114.529: error: Stray end tag "a".
 ..

Expand the below drawer if you’d like to see the full log with 46 errors:

Output of running html5validator --root public/

"file:/public/google4a938eaf9bbacbcd.html":1.1-1.52: error: Non-space characters found without seeing a doctype first. Expected "<!DOCTYPE html>".
"file:/public/google4a938eaf9bbacbcd.html":1.1-1.52: error: Element "head" is missing a required instance of child element "title".
"file:/public/notes/plantuml/index.html":114.474-114.678: error: Attribute "title" not allowed on element "a" at this point.
"file:/public/notes/nim-fmt/index.html":300.34-300.52: error: Duplicate ID "older-issue".
"file:/public/notes/nim-fmt/index.html":306.20-306.33: error: Duplicate ID "floats".
"file:/public/notes/nim-fmt/index.html":339.34-339.52: error: Duplicate ID "older-issue".
"file:/public/notes/nim-fmt/index.html":341.320-341.335: error: Duplicate ID "integers".
"file:/public/notes/nim-fmt/index.html":341.494-341.507: error: Duplicate ID "floats".
"file:/public/notes/nim-fmt/index.html":385.34-385.48: error: Duplicate ID "strings".
"file:/public/notes/string-fns-nim-vs-python/index.html":134.34-134.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":239.34-239.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":269.34-269.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":336.34-336.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":513.106-513.118: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":588.106-588.118: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":636.34-636.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":731.34-731.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":797.34-797.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":846.34-846.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":894.34-894.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":920.34-920.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":942.34-942.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":1012.34-1012.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":1074.34-1074.46: error: Duplicate ID "notes".
"file:/public/notes/nim/index.html":472.480-472.486: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/notes/nim/index.html":1359.221-1359.231: error: Duplicate ID "log".
"file:/public/notes/nim/index.html":2364.130-2364.149: error: Duplicate ID "named-tuples".
"file:/public/notes/nim/index.html":2403.149-2403.172: error: Duplicate ID "anonymous-tuples".
"file:/public/notes/nim/index.html":3370.284-3370.303: error: Duplicate ID "installation".
"file:/public/notes/nim/index.html":3410.354-3410.373: error: Duplicate ID "installation".
"file:/public/notes/nim/index.html":5033.34-5033.52: error: Duplicate ID "older-issue".
"file:/public/notes/nim/index.html":5376.34-5376.45: error: Duplicate ID "json".
"file:/public/notes/nim/index.html":6066.64-6066.81: error: Duplicate ID "references".
"file:/public/bits/plantuml-version/index.html":7.37-7.94: error: An "img" element must have an "alt" attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.
"file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: Start tag "a" seen but an element of the same type was already open.
"file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: End tag "a" violates nesting rules.
"file:/public/auto-count-100daystooffload-posts/index.html":114.526-114.529: error: Stray end tag "a".
"file:/public/generics-not-exactly-in-systemverilog/index.html":118.232-118.238: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/grep-po/index.html":51.139-51.145: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/how-do-i-write-org-mode/index.html":23.194-23.200: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/hugo-use-goat-code-blocks-for-ascii-diagrams/index.html":24.130-24.255: error: An "img" element must have an "alt" attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.
"file:/public/hugo-modules-getting-started/index.html":6.865-6.871: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/using-emacs-advice-to-silence-messages-from-functions/index.html":151.687-151.732: error: Start tag "a" seen but an element of the same type was already open.
"file:/public/using-emacs-advice-to-silence-messages-from-functions/index.html":151.748-151.751: error: Stray end tag "a".
"file:/public/page/6/index.html":29.39-29.54: error: Duplicate ID "fnref:1".
"file:/public/page/6/index.html":49.169-49.184: error: Duplicate ID "fnref:1".

Conclusion #

It was really easy to download the run the vnu application using Java, the standalone Linux binary and also through the html5validator Python wrapper.

After my quick trials, I think I will use the html5validator approach more because,

It works as I expect will the least number of switches.
I am able to redirect the output using html5validator --root public/ > validate.log. I tried the same using the vnu.jar and Linux compiled vnu binary, but the error log redirection didn’t work with those.

Hugo Modules: Importing a Theme

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Thu, 26 May 2022 16:26:00 -0400

A brief guide on how to install Hugo themes using Hugo Modules.

Table of Contents

1 Clean up the old way of setting a theme
2 Import the “theme” module
- Quick Example
3 hugo mod tidy
4 Updating the theme
Dependency Graph
Building your Hugo site on a server
In a nutshell
References

Hello! You are reading this post because you are probably interested in the Hugo Modules feature and are considering to import a Hugo Module as a theme.

Step 0 for that approach is to make your site repo a Hugo Module. If your site already is, then it would have a go.mod file in the repo root. If you don’t have the go.mod file, check out the previous post Hugo Modules: Getting Started first ‼️

If you don’t have a go.mod file for your site repo, and you still decide to continue with the next steps, don’t complain if you see errors like module “foo” not found; either add it as a Hugo Module or store it in “<your site repo>/themes”.: module does not exist. — speaking from experience 😉.

With that out of the way, here are the next steps ..

1 Clean up the old way of setting a theme #

If you are upgrading your Hugo site to switch from the legacy method of using themes (i.e using the theme variable in the site config In my posts, you may have seen me use the Site Config term or config.toml – They mean the same thing. ), you need to clean that up.

Remove the theme variable from your site config.
Remove the themes directory, or move it out of your Hugo site repo.
- If you were cloning a theme developed by someone else in there, you can just remove this directory.
- If you are maintaining your own theme in that directory, move it out of your site repo and convert it to a Hugo Module.

2 Import the “theme” module #

The theme is quoted in this title, because the concept of a Hugo “theme” is a bit old now (<2022-05-26 Thu>) and that has been superseded with the concept of “modules”.

The main difference between a theme and a generic Hugo Module is that the former will allow you to build your site entirely, while the latter might implement only some modular features like enabling the ATOM feed, or adding a search to your website.

I am mentioning this again for convenience, from the previous post in this series:

A module can be your main project or a smaller module providing one or more of the 7 component types defined in Hugo: static, content, layouts, data, assets, i18n, and archetypes. You can combine modules in any combination you like, and even mount directories from non-Hugo projects, forming a big, virtual union file system.

A theme will need to have the “layout” component. Additionally, it might have the “assets”, “static”, and other components too.

Importing a module as a theme will typically look like this in your site config:

[module]
 [[module.imports]]
 path = "URL of the theme's git remote *without* the 'https://' part"

The path here would be something like github.com/USER/THEME-REPO-NAME or gitlab.com/USER/THEME-REPO-NAME.

Note: It’s possible to take any Hugo theme git repo and import that as a Hugo Module even if that repo isn’t actually one i.e. doesn’t have a go.mod. But it’s recommended that the theme be a proper Hugo Module so that you have better dependency tracking between your site and the theme.

Quick Example #

Follow these steps if you want to try out how this Hugo Module based theme importing

As a reminder, you need to have Go installed on your system.

Create a temporary directory somewhere and cd to it.
Initialize your site as a Hugo Module: hugo mod init foo (yeah, type that out literally — it will work)
Create a config.toml file with the below content. It imports the hugo-mwe-theme hugo-mwe-theme is a minimal Hugo theme that I use to quickly try out some new feature in Hugo or to create a minimal working example to reproduce a bug. theme.
```
[module]
 [[module.imports]]
 path = "gitlab.com/kaushalmodi/hugo-mwe-theme"
```
Code Snippet 1: Example of importing a Hugo module as a theme in config.toml
Create content/hello.md. This step is optional and is only so that your test site as some content.
```
+++
title = "Hello"
+++
Hey!
```

That’s it! Now run the Hugo server (hugo server) and look at your site running on localhost .. while thinking in disbelief.. just how easy all of this was! 😃.

Did you need to manually clone any theme? No
Would you need to deal with the .gitmodules file? No

3 `hugo mod tidy` #

Finally, run hugo mod tidy to clean up the go.mod and update/generate the go.sum file. These files will track the module dependencies for your site.

The go.mod contains the direct module dependencies for your site.
The go.sum contains the versions and hashes of all the direct and indirect dependencies Just as you added a theme as a Hugo Module to your site, it’s possible that that theme is depending on other Hugo Modules (like the ones I mentioned earlier: ATOM feeds, search, etc.). for your site.

You would need to commit the go.mod and go.sum¹ files if you build and deploy your website on a remote server or a CI/CD system.

If you ran the Quick Example, you will see this (as of <2022-05-26 Thu>) in your go.mod:

module foo

go 1.18

require gitlab.com/kaushalmodi/hugo-mwe-theme v0.1.1 // indirect

.. and this in your go.sum:

gitlab.com/kaushalmodi/hugo-mwe-theme v0.1.1 h1:FyTp43CJRpBfoHyWnwQFx//cipgP6xQ9/uucj+qjj1U=
gitlab.com/kaushalmodi/hugo-mwe-theme v0.1.1/go.mod h1:vvq0r/SfKMbiPbyqL4YottSOkpCkBSosqGRm82aDNrU=

4 Updating the theme #

Here are some common ways to update the theme module going forward:

Command	Description
`hugo mod get -u`	Update only the modules that your site directly depends on.
`hugo mod get -u ./...`	Update the modules that your site depends on in a recursive fashion.

Additionally, you might or might not need these, but I am documenting them here for completeness:

Command	Description
`hugo mod get -u <module path>`	Update only the specified module² to the latest version. Example: `hugo mod get -u gitlab.com/kaushalmodi/hugo-mwe-theme`
`hugo mod get <module path>@<git ref>`	Update a module to the specified git tag or commit. Example: `hugo mod get gitlab.com/kaushalmodi/hugo-mwe-theme@v0.1.1`

Dependency Graph #

If you have a theme added as a Hugo Module, which depends on other Hugo Modules, it’s often helpful to know the dependency graph. You can do that by running:

hugo mod graph

For the above Quick Example, you will see just this one line because that theme does not depend on other modules:

foo gitlab.com/kaushalmodi/hugo-mwe-theme@v0.1.1

Building your Hugo site on a server #

Alright, so you are able to build your site locally after switching to using themes as modules, great!

Now, if you build and deploy your site on a remote server like Netlify or Vercel, you need to ensure that you have a recent version of Go installed in their environment too.

I deploy this website using Netlify, and so I know how to do that there — Set the GO_VERSION environment variable to a recent version like 1.18 in the Environment variables section in Netlify Build & deploy settings.

In a nutshell #

First convert your Hugo site to a Hugo module.
Then replace the theme in your site config with a module import.

Enjoy! 🍾

References #

It is recommended to git commit the go.sum along with your site’s go.mod. From the Go Modules documention: Ensure your go.sum file is committed along with your go.mod file. See FAQ below for more details and rationale. ↩︎
Trust me.. once you get a hang of the Hugo Module system, your site will have more than one! ↩︎

Posts on A Scripter's Notes

Version controlling Jenkins config

What bit me #

.gitignore for Jenkins config #

Jenkins Plugin Manager #

Full solution #

Using Git Delta with Magit

Installing delta #

Installing magit-delta #

Configuring delta #

Gujarati Transliteration

Enabling transliteration #

Toggling the input method #

Caveats with Gujarati and other Indic language transliteration #

“Input method” cheat sheet #

Closing #

References #

Cleaning up ${GOPATH}/pkg/

The $GOPATH directory #

The disk space issue #

Write-protected ${GOPATH}/pkg/ #

Cleaning up “modcache” #

Wrapping up #

References #

View GitHub Pull Requests in Magit

Locally creating a branch for a PR #

Getting references to all Pull Requests #

Automatically adding PR refs #

Summary #

References #

Gujarati fonts in Emacs

Setting a “fontset” font #

Gujarati fonts #

Emacs Lisp: Advice Combinators

Overview on using advices #

1 :before #

2 :after #

3 :override #

4 :around #

5 :before-while #

6 :before-until #

7 :after-while #

8 :after-until #

9 :filter-args #

10 :filter-return #

Summary #

References #

Org: Show only Post subtree headings

Org Global Cycle #

Skeleton of only Post headings #

The “Collapse All Posts” function #

Binding with C-u C-c TAB #

Result #

Firefox: Always open a New Tab after Current

The Issue #

Expected Behavior #

False Leads #

Firefox Bug # 1485683 #

Solution #

Saving Python pip dependencies

1 Create a virtual environment #

2 Activate the virtual environment #

3 Install the packages using pip #

4 Generate requirements.txt #

Netlify deployment #

Summary #

Disarming the 'tar' bomb in 10 seconds

Mnemonics #

Creating an archive #

Extracting an archive #

Listing contents of an archive #

Summary #

Unclutter: A better Reader View for browsers

Why? #

Caveat #

Closing #

Zero HTML Validation Errors!

1 Avoid duplicate heading id attributes #

2 Remove inline <style> elements #

3 Ensure that all images have captions or alt attributes #

`.gitignore` for Jenkins config #

Installing `delta` #

Installing `magit-delta` #

Configuring `delta` #

The `$GOPATH` directory #

Write-protected `${GOPATH}/pkg/` #

1 `:before` #

2 `:after` #

3 `:override` #

4 `:around` #

5 `:before-while` #

6 `:before-until` #

7 `:after-while` #

8 `:after-until` #

9 `:filter-args` #

10 `:filter-return` #

Binding with `C-u C-c TAB` #

3 Install the packages using `pip` #

4 Generate `requirements.txt` #

1 Avoid duplicate heading `id` attributes #

2 Remove inline `<style>` elements #

3 Ensure that all images have captions or `alt` attributes #

Using the `.jar` #

Using `pip install` #

3 `hugo mod tidy` #