git on A Scripter's Notes

Version controlling Jenkins config

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Wed, 20 Jul 2022 00:18:00 -0400

Jenkins is an amazing free and open source continuous integration and deployment software. But its primary means of configuration is a web UI, and recently that cost me a lot of debug time. That set me down the path of figuring out a way to version control the Jenkins config (the $JENKINS_HOME directory).

Table of Contents

What bit me
.gitignore for Jenkins config
Jenkins Plugin Manager
Full solution

Jenkins is a wonderful piece of software and I use it with my Bitbucket git repos for CI/CD.

Jenkins uses a web UI for its configuration. I dislike that because it’s difficult to document the configuration process without screenshots, and if I need to create a new server, it’s a manual process of clicking through tabs and filling in the text boxes. I didn’t mind this enough to do anything about it .. that is until I finally got bit by it.

What bit me #

Without going into too much detail, that issue was multi-fold:

I had unknowingly messed up the Project-based Matrix Authorization Strategy such that other users in my team were not able to view the Jenkins jobs.
I had also updated the Jenkins server that introduced a bug (JENKINS-68748) where the Test LDAP Settings failed with an error, but the LDAP authentication actually worked!
I had also updated all the plugins after updating Jenkins. So if I rolled back the Jenkins versions, most of the plugins would fail because of incompatibility with the older Jenkins version. I had updated Jenkins after months!

That’s when I wished that my whole Jenkins was version-controlled. That would have allowed me to roll back to the last working “Jenkins image” with the Jenkin version, plugins' versions and my Jenkins config all in sync.

I had delayed doing this because my $JENKINS_HOME was more than 1GB in size and I didn’t have time or motivation to figure out what stuff I should commit and what I should ignore .. But no more — The time had finally come.

`.gitignore` for Jenkins config #

So I did what any good engineer would do .. start looking for a solution online. I found this StackOverflow answer for Is there a way to keep Hudson / Jenkins configuration files in source control?.

That answer shares a .gitignore that ignores files not necessary for configuring a Jenkins server — Example: job builds, workspace, log files, etc. But it didn’t work out of the box because the plugin version info wasn’t getting committed correctly. I had committed everything to git after using the suggested .gitignore and pushed to my git remote. But if I cloned that repo to a different area and attempted to start the Jenkins server from there, it crashed with this message:

2022-07-15 13:07:06.082+0000 [id=31] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading global config
com.thoughtworks.xstream.mapper.CannotResolveClassException: hudson.security.ProjectMatrixAuthorizationStrategy
 at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:81)
 at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
 at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55)
 at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
 at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88)
 at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
 at com.thoughtworks.xstream.mapper.ClassAliasingMapper.realClass(ClassAliasingMapper.java:79)
 at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
 at com.thoughtworks.xstream.mapper.ArrayMapper.realClass(ArrayMapper.java:74)
 at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
 at com.thoughtworks.xstream.mapper.SecurityMapper.realClass(SecurityMapper.java:71)
 at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
 at hudson.util.XStream2$CompatibilityMapper.realClass(XStream2.java:411)
 at hudson.util.xstream.MapperDelegate.realClass(MapperDelegate.java:46)
 at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
 at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:47)
 at hudson.util.RobustReflectionConverter.determineType(RobustReflectionConverter.java:521)
 at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:346)
Caused: jenkins.util.xstream.CriticalXStreamException:
---- Debugging information ----
cause-exception : com.thoughtworks.xstream.mapper.CannotResolveClassException
cause-message : hudson.security.ProjectMatrixAuthorizationStrategy
class : hudson.model.Hudson
required-type : hudson.model.Hudson
converter-type : hudson.util.RobustReflectionConverter
path : /hudson/authorizationStrategy
line number : 12
version : not available
-------------------------------

Code Snippet 1: Snippet of Jenkins crash when attempting to run the server from the freshly cloned git repo

Jenkins Plugin Manager #

So I reached out for help on the Jenkins Community. One of the key contributors to Jenkins, Mark Waite, was tremendously helpful. He suggested using his jenkins-plugin-manager tool. After trying it out for a bit, I realized that this tool had everything I needed for version controlling the plugin versions:

Ability to save a list of installed Jenkins plugins and their versions to a file.
Ability to batch install all the plugins of the versions listed in a file.

This was like doing Python’s plugin management using requirements.txt, except that this was for Jenkins.

Full solution #

With a combination of the .gitignore that I started with from that SO answer, managing plugins using jenkins-plugin-manager, tweaking the .gitignore to my liking, and adding helper Bash scripts for downloading and running Jenkins server binaries, and doing the plugin management, I finally got what I needed:

https://github.com/kaushalmodi/jenkins-minimal

The README on the repo has all the instructions.

Using Git Delta with Magit

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Wed, 06 Jul 2022 22:04:00 -0400

Git Delta is a command line utility that beautifies git diffs in the terminal. But did you know that it can do the same in Magit too?

Table of Contents

Installing delta
Installing magit-delta
Configuring delta

Delta is a highly configurable I am not kidding. Check out the output of delta --help. command line utility that makes the git diffs look better, while also syntax-highlighting Difftastic is another popular diff tool which compares files based on their syntax. I like reviewing git diffs from within Emacs (Magit). But difftastic does not support Magit. the code in the diffs.

When I first heard of “syntax highlighted diffs”, I wasn’t sure what that meant. If you are in the same boat, here’s a screenshot that shows that.

Figure 1: Example of how delta renders a git diff for an ox-hugo commit

But I do most of my git operations including viewing of diffs from within Emacs, using Magit.

.. and thankfully delta works with Magit!

Below screenshot shows how the same diff looks like in Magit.

Figure 2: Example of how magit-delta renders a git diff for an ox-hugo commit

The magit-delta Emacs package makes this possible, which is also developed by the delta author Dan Davison.

Caveat: If the line numbers are enabled in delta, they mess up the interactive expanding and collapsing of diffs in Magit. See Magit Delta Issue # 13 for more details.

Now, I am alright with not seeing the line numbers in Magit. But I really liked to see the line numbers in the side-by-side view in the terminal. Luckily, if disabled the line-numbers feature but enabled the side-by-side view, I got what I wanted!

Line numbers are disabled in Magit and expanding/collapsing of diffs works correctly. I am also really glad that I don’t see the side-by-side view in Magit diffs even when I enable that feature in delta, because I like to have my Emacs buffers only about 90 characters wide.
Line numbers and side-by-side view are enabled in the terminal.

I’ll end this post with pointers to installing delta and magit-delta and how to configure them.

Installing `delta` #

You can install delta (it’s called git-delta in some package managers) using one of the methods listed in its manual, or you can download → extract its statically compiled binary for your OS from its GitHub Releases page.

Installing `magit-delta` #

Once you put this snippet in your Emacs config and evaluate it, it will install this package and enable the magit-delta-mode in the Magit buffers.

(use-package magit-delta
 :ensure t
 :hook (magit-mode . magit-delta-mode))

Code Snippet 1: Installing and enabling magit-delta using use-package

Configuring `delta` #

Here’s a snippet for delta configuration from my .gitconfig. It’s mostly the same as the one in delta’s the Getting Started guide. The main difference in my workaround for the magit-delta issue.

[core]
 pager = delta

[interactive]
 diffFilter = delta --color-only
[add.interactive]
 useBuiltin = false # required for git 2.37.0

[diff]
 colorMoved = default

[delta]
 # https://github.com/dandavison/magit-delta/issues/13
 # line-numbers = true # Don't do this.. messes up diffs in magit
 #
 side-by-side = true # Display a side-by-side diff view instead of the traditional view
 # navigate = true # Activate diff navigation: use n to jump forwards and N to jump backwards
 relative-paths = true # Output all file paths relative to the current directory
 file-style = yellow
 hunk-header-style = line-number syntax

Code Snippet 2: My configuration for delta in .gitconfig

View GitHub Pull Requests in Magit

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Thu, 23 Jun 2022 17:51:00 -0400

How to view GitHub Pull Request branches locally in the cloned repo, and more importantly, how to do that automatically from within Emacs.

Table of Contents

Locally creating a branch for a PR
Getting references to all Pull Requests
Automatically adding PR refs
Summary
References

I have a few public projects in git repos, but I don’t get that much traffic in Pull Requests (PR) Gitlab calls these Merge Requests or MRs. . So when I need to add additional commits to a PR, I would just add the PR author’s remote to my local repo, check out their PR branch, add my own commits and then merge that to my project’s main branch.

As these occurrences were few and far apart, I didn’t have a need to view the PR branches directly from within Emacs/Magit. Though, I somehow knew that each GitHub Pull Request’s HEAD got assigned a git reference. But I didn’t need to use that knowledge until today 😃.

Today, when discussing PR # 20 on Prot’s Denote package’s GitHub mirror Prot uses SourceHut as the primary git forge for his Emacs packages. But I am glad that he doesn’t mind the activity in Issues and Pull Requests on the GitHub mirror. , he wrote this comment:

Now I just need to figure out how best to incorporate your changes into the org-id branch so I can add the final bits. I am not too familiar with the PR workflow …

.. and that inspired this post today.

Locally creating a branch for a PR #

From the GitHub docs, the git command to create a local branch for a PR is this:

git fetch origin pull/ID/head:BRANCHNAME

I have the denote package cloned from its GitHub mirror. So the origin remote’s url is https://github.com/protesilaos/denote.

Make sure that the remote name used in this command is pointing to a GitHub repo, and not a mirror forge like GitLab or SourceHut.

When I ran the below command, I got a new branch pr-20 pointing to the latest commit of that PR:

git fetch origin pull/20/head:pr-20

Awesome!

.. But that wasn’t good enough
.. Now I wanted more
.. I didn’t want to manually create a branch for each PR.

Getting references to all Pull Requests #

Now that I was on that quest of “I want more”, it didn’t take me long to re-discover this 7-year old nugget by Oleh Krehel. Here are the relevant bits from that post:

Open the local repo’s .git/config file.
Find the [remote "origin"] section

Modify it by adding this one line with pull refs. This is the same for all GitHub repositories.

[remote "origin"]
 url = https://github.com/USER/REPO.git
 fetch = +refs/heads/*:refs/remotes/origin/*
 fetch = +refs/pull/*/head:refs/pull/origin/*

With that edit in place, when I did l a (show the log for all git references), followed by f a (fetch all the remotes) in the Magit, I could see the references to the denote repo’s PRs!

Figure 1: Viewing PR references from denote package’s GitHub repo

.. But that still wasn’t good enough
.. I didn’t want to manually edit the .git/config in each repo.

Automatically adding PR refs #

Of course, I wasn’t the first one to think of this!

Another Emacs veteran Artur Malabarba had already had this covered also around 7 years back. Coincidentally, that post was written as a response to that same blog post by Oleh where he shared the above .git/config tip.

In that post, Artur shares an Emacs Lisp function that uses Magit functions like magit-get, magit-get-all and magit-git-string to auto-add the fetch = +refs/pull/*/head:refs/pull/origin/* line in the .git/config. This magic happens after checking that the origin remote points to a GitHub repo, and if that line doesn’t already exist.

Here, I am lightly modifying the function shared in that post so that the origin remote name is not hard-coded The reason is that sometimes, I name the original remote as upstream and my fork as fork, and I might have no remote named origin. . Credit for the main logic in this code still goes to Artur.

(defun modi/add-PR-fetch-ref (&optional remote-name)
 "If refs/pull is not defined on a GH repo, define it.

If REMOTE-NAME is not specified, it defaults to the `remote' set
for the \"main\" or \"master\" branch."
 (let* ((remote-name (or remote-name
 (magit-get "branch" "main" "remote")
 (magit-get "branch" "master" "remote")))
 (remote-url (magit-get "remote" remote-name "url"))
 (fetch-refs (and (stringp remote-url)
 (string-match "github" remote-url)
 (magit-get-all "remote" remote-name "fetch")))
 ;; https://oremacs.com/2015/03/11/git-tricks/
 (fetch-address (format "+refs/pull/*/head:refs/pull/%s/*" remote-name)))
 (when fetch-refs
 (unless (member fetch-address fetch-refs)
 (magit-git-string "config"
 "--add"
 (format "remote.%s.fetch" remote-name)
 fetch-address)))))
(add-hook 'magit-mode-hook #'modi/add-PR-fetch-ref)

Code Snippet 1: Function to auto-add GitHub PR references to the repo's .git/config

Summary #

With the above snippet added to your Emacs config and evaluated, each time you visit a repo cloned from GitHub in the Magit Status buffer (M-x magit-status), the PR refs will get auto-added to that repo’s .git/config if needed.

After that, you can easily view the commits from all the PRs by doing l a f a.

References #

Creating a patch file using Magit

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Sun, 08 May 2022 08:48:00 -0400

Quick tip on how to create git patch files in Emacs using Magit.

Table of Contents

Single-file patch
Multi-file patch
More Resources

Recently I came across few instances where people were asking questions related to creating patches for contributions to Emacs and Org mode repos here and then here. I was in the same shoes back then when I was about to make my first contribution to Emacs. And so I thought of sharing this tip on how to use Magit to create patch files.

If you have been using both Emacs and git, you might have already heard about the awesome Magit package. If you haven’t 😲, check out this screenshot-annotated review of what Magit is. With that out of the way, and assuming that you already have it installed (use-package magit :ensure t) , here’s how to create a patch file using Magit ..

Single-file patch #

Commit your changes to the git repo first.
Bring up the Magit Log view. From the Magit status buffer, you would type l l to show the log of the current branch.
Move the point to the commit that you want to send as a patch file, and hit W c c RET.
- The last RET selects the commit the point is on, in the ∗magit-log∗ buffer.
- If the first line of the commit log of the selected commit is “Update docstrings for shortdoc.el”, you’ll see a patch file named 0001-Update-docstrings-for-shortdoc.el.patch created in your git repo root.
- You can now email this patch file as an attachment to bug-gnu-emacs@gnu.org (if contributing to Emacs) or to emacs-orgmode@gnu.org (if contributing to Org mode).

Multi-file patch #

If you need to create a multi-file patch i.e. patch files for a series of commits, select those commits in the ∗magit-log∗ buffer The commit selection process is the same as how you would select text in any Emacs buffer. For example, if I want to create a series of 5 patches, I would go to the latest commit in the series, hit C-SPC and then C-n 4 times to select 5 rows of commits. , and then use the same W c c binding.

More Resources #

Here are the official contribution guides for Emacs and Org mode:

Here are some more resources that got shared in the Emacsverse recently (within the past year as of writing this):

<2022-04-23 Sat> Contributing patches to Org – Ihor Radchenko
<2022-04-09 Sat> Primer on formatting Git patches with Emacs (Magit) – Protesilaos Stavrou
<2021-08-17 Tue> Contributing to Emacs – Colin Woodbury

Git diff Minified JS and CSS

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Mon, 19 Mar 2018 18:13:00 -0400

Make the git diff output be more useful when diffing minified .js and .css files.

Table of Contents

1 Install js-beautify using npm
2 Configure git to use that tool
3 Add/update .gitattributes file to the project repo
Beautiful Result

While working on a little PR for the Hugo doc site theme, I learned that if I needed to make changes to JS/CSS, I had to commit my changes in both unminified and minified versions.

I have a habit to always look at the diffs at the time of staging and committing. So it felt very unnatural to commit a minified JS where the diff would show just one changed line with thousands of characters of minified+uglified JS.

So I started looking for solutions, and found this post by Christian Weiske where he suggests using js-beautify to beautify minified JS diffs.

And that tool works beautifully!

I later found out that the same tool can also be used to beautify minified CSS.
.. and I installed that tool using npm as the pip3 approach failed with “Collecting js-beautify.. Could not find a version that satisfies the requirement js-beautify (from versions: ) No matching distribution found for js-beautify”.

So here’s how you can do useful git diff for minified JS and CSS.

1 Install `js-beautify` using `npm` #

I see myself using js-beautify in many other projects too. So I installed it globally.

npm install --global js-beautify

2 Configure `git` to use that tool #

Add below to your ~/.gitconfig:

Use js-beautify to first beautify the minified JS for the minjs diff configuration, and then do a diff of those beautified files.
Enable caching of those beautified files to speed up the diff, so that re-beautification of unmodified minified files can be skipped.
Similarly for minified CSS, use js-beautify --css to first beautify the minified CSS for the mincss diff configuration.

[diff "minjs"]
 textconv = js-beautify
 cachetextconv = true
[diff "mincss"]
 textconv = js-beautify --css
 cachetextconv = true

3 Add/update `.gitattributes` file to the project repo #

Now, in your project repo’s .gitattributes file, you need to associate files with the diff configurations set above.

Below will use the minjs configuration for *.min.js and *.bundle.js files, and mincss configuration for *.min.css and main.css.

*.min.js diff=minjs
*.bundle.js diff=minjs
*.min.css diff=mincss
main.css diff=mincss

Beautiful Result #

Figure 1: git diff of minified JS as seen in Magit

Isn’t that better than how GitHub shows the exact same commit diff? 😎

Figure 2: Same commit diff shown on GitHub

Narrowing the Author column in Magit

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Mon, 18 Dec 2017 16:36:00 -0500

The Org and Worg git repos can be committed to only via ssh protocol (and not https), and for technical reasons, I cannot commit to those repos via ssh from my primary development machine. So I ended up with a flow that involves pushing commits to those repos using my Nexus 6p phone.

Magit is how I git, and I love to primarily work from the ∗magit-log∗ buffer (M-x magit-status, l b). But the default column widths were not optimal on a Nexus 6p 5.7" screen. And that’s what inspired this tweak, which looks great on a regular desktop monitor too.

Before I jump to the code, you can see in the below figure how the author and commit age columns took up roughly half the screen width on my phone before the tweak (left), and how much easier it is to read the commit messages after the tweak (right).

Figure 1: Org-mode Worg commit log : Before (Left), After (Right)

So the Magit Log columns tweaks are basically:

Abbreviate the authors’ first names to just their initials.
Abbreviate the commit ages to 1-character time units.

To implement those tweaks, I started digging through magit-log.el and found that I would need to customize (i) the magit-log-margin variable and (ii) magit-log-format-margin function.

Customizing magit-log-margin variable allows me to use the abbreviated age instead of the verbose age strings (minute→m, hour→h, day→d, week→w, month→M, year→Y). It’s also where I specify how wide I want the author column to be.. I can now reduce that column width from the default value of 18 to 11 as I am abbreviating the author name too.
Reviewing the magit-log.el code, I realized that I also need to advise the magit-log-format-margin function to implement the author name abbreviation.

In the below code snippet, I am using use-package¹ to customize that variable and function in magit-log.

(use-package magit-log
 :init
 (progn
 ;; (setq magit-log-margin '(t age magit-log-margin-width t 18)) ;Default value
 (setq magit-log-margin '(t age-abbreviated magit-log-margin-width :author 11)))
 :config
 (progn
 ;; Abbreviate author name. I added this so that I can view Magit log without
 ;; too much commit message truncation even on narrow screens (like on phone).
 (defun modi/magit-log--abbreviate-author (&rest args)
 "The first arg is AUTHOR, abbreviate it.
First Last -> F Last
First.Last -> F Last
Last, First -> F Last
First -> First (no change).

It is assumed that the author has only one or two names."
 ;; ARGS -> '((REV AUTHOR DATE))
 ;; (car ARGS) -> '(REV AUTHOR DATE)
 ;; (nth 1 (car ARGS)) -> AUTHOR
 (let* ((author (nth 1 (car args)))
 (author-abbr (if (string-match-p "," author)
 ;; Last, First -> F Last
 (replace-regexp-in-string "\\(.*?\\), *\\(.\\).*" "\\2 \\1" author)
 ;; First Last -> F Last
 (replace-regexp-in-string "\\(.\\).*?[. ]+\\(.*\\)" "\\1 \\2" author))))
 (setf (nth 1 (car args)) author-abbr))
 (car args)) ;'(REV AUTHOR-ABBR DATE)
 (advice-add 'magit-log-format-margin :filter-args #'modi/magit-log--abbreviate-author)))

NOTES

I need to set the magit-log-margin value in the :init block because that variable dynamically sets many other variables in magit-log at the time of loading. So we cannot wait for the whole magit-log to load before setting that variable. I wouldn’t need to worry about this if I were using the Emacs Customize interface to set this variable.
I use modi/magit-log--abbreviate-author to add a :filter-args advice to magit-log-format-margin. The signature of the advised function is (magit-log-format-margin REV AUTHOR DATE) (as of Magit 20180809.1716). So the advice basically replaces the AUTHOR arg with its abbreviated form as explained in the modi/magit-log--abbreviate-author doc-string.

Feel free to ask for more explanation in comments in the case you don’t use use-package. ↩︎

git on A Scripter's Notes

Version controlling Jenkins config

What bit me #

.gitignore for Jenkins config #

Jenkins Plugin Manager #

Full solution #

Using Git Delta with Magit

Installing delta #

Installing magit-delta #

Configuring delta #

View GitHub Pull Requests in Magit

Locally creating a branch for a PR #

Getting references to all Pull Requests #

Automatically adding PR refs #

Summary #

References #

Creating a patch file using Magit

Single-file patch #

Multi-file patch #

More Resources #

Git diff Minified JS and CSS

1 Install js-beautify using npm #

2 Configure git to use that tool #

3 Add/update .gitattributes file to the project repo #

Beautiful Result #

Narrowing the Author column in Magit

`.gitignore` for Jenkins config #

Installing `delta` #

Installing `magit-delta` #

Configuring `delta` #

1 Install `js-beautify` using `npm` #

2 Configure `git` to use that tool #

3 Add/update `.gitattributes` file to the project repo #