As I mentioned in the post description above, the solution to clean up the auto-populated ${GOPATH}/pkg/ is to run go clean -modcache. While the solution is simple, it wasn’t easy for me to discover, and so I am writing this short piece to make it a bit more discoverable for others like me.

The $GOPATH directory #

The GOPATH environment variable lists places to look for Go code.

If you haven’t set this variable and if the ${HOME}/go/ directory isn’t used to contain the Go distribution, $GOPATH defaults to that path.

When you run go get to install any Go package,

• ${GOPATH}/pkg/ gets populated with all the Go module dependencies,
• .. and all package’s executables if any get installed to ${GOPATH}/bin/.

To confirm the value of GOPATH used by Go, see go env | rg GOPATH.

The disk space issue #

I have been happily installing and building Go apps like Hugo using go get or go build, and all these installations would end up in the $GOPATH. But over time, I noticed that the disk space used by ${GOPATH}/pkg/ just kept on creeping up.

Today I happened to notice that this directory was taking up roughly 4 GB of my disk space! I started analyzing why it was taking up so much space using my favorite tool for this purpose – ncdu.

Here’s a snapshot showing disk usage by one of the sub-directories under ${GOPATH}/pkg/, which shows the problem — Over time, I had accumulated multiple versions of multiple packages!

Write-protected ${GOPATH}/pkg/ #

I identified the problem. And I knew that I just needed to delete all the old packages.

I had been searching a solution to this on and off, but didn’t have much success, mainly attributed to the short and generic name of the “Go” language, and the fact that I didn’t know what the ${GOPATH}/pkg/ directory was called.

Cleaning up “modcache” #

Today, I finally had success when I searched for this magic phrase: golang “cannot remove” “pkg/mod” .. and Golang Issues # 27161 was the first search result!

The solution was so simple, but so difficult to look for ..

go clean -modcache

From that issue, I also learned that the ${GOPATH}/pkg/ directory is the default “modcache” or the cache directory for holding all the installed Go modules.

Wrapping up #

Knowing that the stuff I was trying to delete is called modcache, this of course works ..

go help clean | rg modcache -A 2

The issue referenced above gets resolved when a -modcacherw switch gets added to the go build command. I see that switch when I run go help build. But that switch is not available for go get? .. because I don’t see it when I run go help get.

I don’t understand why Go decided to make this so complicated by taking away the write access from the user who installed the Go package!

References #

• go Documentation – GOPATH environment variable
• go Documentation – go help clean
• Go Wiki – Setting GOPATH
• Golang Issue # 27161

Today on Hugo Discourse, I came across the – Why “2006”? – question regarding the use of {{ now.Format "2006" }} in Hugo template. That template simply prints the current year i.e. the time when that template was rendered by Hugo.

As I was answering that, I thought that this was a good time to document this for myself too.. and thus this post.

Time.Format Syntax #

Hugo uses the Go Time.Format method¹ for formatting date and time strings. But that format specification uses a strange syntax — You need to associate 1, 2, 3, 4, 5, 6, 7 with date elements as follows:

You can visualize a formatting string "Jan 2 15:04:05 2006 MST" as below:

Jan 2 15:04:05 2006 MST
  1 2  3  4  5    6  -7

Hugo’s dateFormat #

The dateFormat accepts a date/time format using the same syntax.

Here is the signature of that function:

dateFormat "<FORMAT_STRING>" "<DATE_RFC3339>"

• The <FORMAT_STRING> uses one or more date/time fields as shown in the “Field Strings” column in Table 1. Few examples:
  • “Jan 2 15:04:05 2006 MST”
  • “2006-01-02”
  • “Jan 2, Mon”
  • “It’s 3 O’clock”
• The <DATE_RFC3339> is a date string compatible with RFC3339. Few example valid date strings:
  • “2017-07-31”
  • “2017-07-31T17:05:38”
  • “2017-07-31T17:05:38Z”
  • “2017-07-31T17:05:38+04:00”
  • “2017-07-31T17:05:38-04:00”

Ordinal dates #

In addition, humanize can be used to generate ordinal date, like “1st”, “2nd”, “3rd”, ..

Examples #

References #

• gohugohq.com – Hugo dateFormat
• Hugo documentation – dateFormat
• f***inggodateformat

First of all—I get it. Golang is not the only language that has this odd behavior related to octals. But following the foot-steps of ancestor languages in this particular aspect does not mean that Golang is doing the Right Thing(TM).

I got many interesting comments at:

• A thread on r/programmingcirclejerk
• A thread on r/golang
• HN thread
• And on the original post itself.

Here are some highlights.

zcdopizocioics from Hacker News

The strconv function simply acts in the same manner as the language does, see https://play.golang.org/p/rg0VwhCDVB_C. The idea for strconv originates in c (see atoi etc), and c also has 0-octal notation. Go is simply mimicing c in this regard.

My reply – I just wished the octals needed a stricter prefix like “0o” as they are not very common compared to decimal and hex (in my experience at least).

karmakaze from Hacker News

A better solution is to always provide the base (e.g. 10) rather than using 0 if you don’t want special interpretation of ‘0x’ and ‘0’ prefixes. That should be in Hugo. The problem is not with the golang library which was well documented and always worked as such. By placing any blame on the golang library, there’s less incentive to fix Hugo.

My reply – That’s a valid point. I have made that suggestion to the Hugo devs. Someone on Reddit also suggested using strconv.Atoi() instead of ParseInt(). I have added to the suggestion too.

/u/shekelharmony from Reddit

This isn’t really a quirk of Go - the documentation says exactly how strconv.ParseInt() works, and you would only get that behavior by intentionally setting the second argument to 0. If you’re always working with decimals, you might use strconv.Atoi() instead since it only takes one argument. You could say it’s a quirk of Hugo though, since it sounds like they don’t have a function for parsing decimals at all.

My reply – .. I’ll definitely forward your suggestion with the Hugo dev team. (I then do that, which I linked above.)

/u/FascinatedBox from Reddit

Java and C both have this and it’s incredibly stupid. Every once in a while some noob thinks that leading zeroes don’t matter (they shouldn’t). So they write a number like 0123 and it doesn’t come out to 123 but there is nothing you can look up to figure out why. There’s no function you’re calling to say “okay, why is function X doing this transformation?” It’s really not hard to have some sort of a prefix. We have a prefix for binary numbers when languages have that. We have one for hex. Why not octal. My language uses 0c to denote octal so that people don’t accidentally fall into this trap. I added that in precisely because I saw someone in a programming class get burned by accidental octal. I knew what the problem was right away, because I’m a PL nerd. But a lot of people don’t want octal, and they certainly aren’t expecting it.

My reply – Thank you. Exactly my point.. We have prefixes for binary and hexadecimal. So I find it odd that octal is given a special treatment by it not needing a prefix! And Golang was created in 2009; it shouldn’t have blindly ported that from C. And who IS writing octal literals today, so furiously, in so much abundance, that they cannot afford using a normal prefix like 0o? Kudos to Python for obsoleting the 0 octal prefix in lieu of 0o in Python 3.

/u/stone_henge from Reddit

poor guy, he just goes on and on about it forever, no way for anyone to step in in the middle of it and point out to him that he’s being dumb.

My reply – Yup. Poor, dumb, me.

/u/Noughmad from Reddit

The handling of octal numbers in C is stupid, I have been bitten by it especially when trying to align multiple numbers with a different number of digits. There should be some non-digit identifier (and not O because it looks to much like 0 in a fixed-width font) used for specifying octal numbers, like x for hexadecimal and now b for binary. But once again, C gets a pass because it was made over 40 years ago. Go doesn’t have this excuse.

My reply – Thank you. Exactly.

/u/pingpong from Reddit

Relevant golang bugs, where people try to fix octal: golang issue #22671, golang issue #151, golang issue #12711.

My reply – Great! So I am not alone.

You can enjoy more of such comments by visiting the Reddit thread linked above.

I would though welcome constructive comments below, or if you prefer, at one of those reddit/HN threads linked above.

Conclusion #

Golang is a modern language born in 2009. It could have fixed this by requiring a more logical prefix like 0o for Octals. Or can it still?

At least I hope that parsing of numbers in front-matter in Hugo gets fixed as it’s unlikely for folks to use octal numbers there.

The aim of this post is to make a Golang quirk more of a common knowledge, with an ulterior motive to eventually get it fixed upstream, somehow..

From my perspective, I see strconv as an internal Go package that any (most of?) Go coder would use to do string → int conversions. If so, I don’t grasp the rationale behind why the strconv developers would make this strange decision.. strange because in normal languages like Python, int("010") returns 10.

Problem #

I learned about this issue for the first time from this Hugo Discourse thread. The synopsis is that someone is retrieving US city zip-codes from a Hugo front-matter variable, and then using some conditional logic based on the last 2 digits.

So the code was:

<!-- Value of .Params.cityZipCode is "75009" -->
{{ if (int (last 2 .Params.cityZipCode)) eq 1 }}er{{ else }}e{{ end }}

The logic is simple.. Get the last two characters of .Params.cityZipCode, which would be "09", convert that string to a number (int), and check if it is 1.

But of-course that didn’t work:

unable to cast “09” of type string to int

Cause #

Later, as I learn, that’s because of the ParseInt function from the strconv package. There it says (emphasis mine):

func ParseInt(s string, base int, bitSize int) (i int64, err error)

ParseInt interprets a string s in the given base (0, 2 to 36) and bit size (0 to 64) and returns the corresponding value i.

If base == 0, the base is implied by the string’s prefix: base 16 for "0x", base 8 for "0", and base 10 otherwise. For bases 1, below 0 or above 36 an error is returned.

Workaround #

This led me to update the int function documentation for Hugo with an ugly workaround:

{{ int ("00987" | strings.TrimLeft "0") }}

Resurgence #

This problem of number-strings beginning with “0” considered as octals resurfaced recently in Hugo issue #4628.

Though, the reported error did not make it evident that that was the problem:

INFO 2018/04/15 18:49:36 found taxonomies: map[string]string{"category":"categories", "manufacturerletter":"manufacturerletters", "manufacturer":"manufacturers", "featured":"featured", "tag":"tags"}
panic: interface conversion: interface {} is float64, not int

goroutine 50 [running]:

github.com/gohugoio/hugo/hugolib.(*Site).assembleTaxonomies(0xc4204ce2c0)
	/go/src/github.com/gohugoio/hugo/hugolib/site.go:1545 +0xee2

The issue reporter had 1500+ content files, and one or more of those files caused this uncaught exception (which is a separate issue, and is planned to be fixed in Hugo) to happen. So I had to spend quite some time doing “forensic debug”¹ to understand what caused that “interface conversion: interface {} is float64, not int”.

Debug #

• The exception was thrown at this line (highlighted below):

  for _, p := range s.Pages {
      vals := p.getParam(plural, !s.Info.preserveTaxonomyNames)
      weight := p.getParamToLower(plural + "_weight")
      if weight == nil {
          weight = 0
      }
      if vals != nil {
          if v, ok := vals.([]string); ok {
              for _, idx := range v {
                  x := WeightedPage{weight.(int), p}
  

• So it was evident that one of the taxonomy weight (manufacturers_weight in this case) values wasn’t getting casted to int.

• So I grepped for anything non-int in those values, like ., ,, e or E, but found nothing.

• Then doing rg ':\s[0-9]{7,}(\.[0-9]+)*$' in the content files, I saw that there were 4 files that had oddly high weight values like 4611000, and wondered if that was somehow the problem. But that wasn’t it either.

• When I deleted all the manufacturers_weight lines in those 1500+ files, the error went away.

  find . -name "*.md" -print0 | xargs -0 sed -i '/manufacturers_weight:.*/d'
  

• So then I restored all of those deleted lines, and started deleting them again, this time in progression ..

  • First deleting all the lines with values with 7 or more digits.. Error still present.
  • Then deleting all lines with values with 6 digits.. Error still present.
  • .. Error still present.
  • Finally when I deleted all lines with values with 4 digits, the error went away!
  • But by now, I had modified about 700 files in this process!

• I had almost given up on debugging this further, when I decided to give the git diff one last glance.. and I found the pattern..

      .. the freaking leading 0’s in some of those manufacturers_weight values!
  

I had a strong gut feeling that those zeros were the problem. So I once again restored the deleted lines in all the content files, typed out the below² with confidence ..

find . -name "*.md" -exec grep -P 'manufacturers_weight: 0[0-9]+' -l {} \; -exec sed -r -i 's/(manufacturers_weight: )0([0-9]+)/\1\2/' {} \;

This ended up with just 16 modified files with a diff like this:

...
modified   content/movements/b/buren/buren-04.en.md
@@ -12,7 +12,7 @@ image: "Buren_04.jpg"
 movementlistkey: "buren"
 caliberkey: "04"
 manufacturers: ["buren"]
-manufacturers_weight: 04
+manufacturers_weight: 4
 categories: ["movements","movements_b","movements_b_buren_en"]
 widgets:
   relatedmovements: true
modified   content/movements/c/citizen/citizen-0153.de.md
@@ -12,7 +12,7 @@ image: "Citizen_0153.jpg"
 movementlistkey: "citizen"
 caliberkey: "0153"
 manufacturers: ["citizen"]
-manufacturers_weight: 0153
+manufacturers_weight: 153
 categories: ["movements","movements_c","movements_c_citizen"]
 widgets:
   relatedmovements: true
...

So that provided that issue originator a workaround so that they can at least get their site built.

But I hope that this 0-leading octal absurdity gets fixed at the root level — People should once again say with confidence, as they learned as kids, that “010” is the same thing as “10”.

Next Steps? #

• Hugo fixes this issue (4628) on its end by not making this exception go uncaught, and instead let the user know that they magically added a non-int-castable octal value in their content in X file on Y line.

• The Golang team gives some serious thought to this stupid (sorry about that) annoying decision:

  If base == 0, the base is implied by the string’s prefix: base 16 for "0x", base 8 for "0" ..

There are two reasons why I suggest installing go to anyone, whether they are Go developers, or not (like me).

1. You can then build amazing utilities like peco, hugo and noti.
2. It’s easy!

Installing go #

Below instructions are for installing go on a 64-bit GNU/Linux machine, and using tcsh shell. But similar steps should work for any other OS and shell.

1. Download the tar.gz for the latest linux-amd64 binaries from https://golang.org/dl/.
2. Extract it to some place in your $HOME. I extract it to ${HOME}/go/¹.
3. Create a directory where you would want to install the go packages.

   mkdir -p ~/go.apps
   

4. Set the following environment variables², and also save them to your shell config:

   setenv GOROOT ${HOME}/go # go root
   setenv GOPATH ${HOME}/go.apps # for go applications
   

5. Add the ${GOROOT}/bin and ${GOPATH}/bin directories to your $PATH.

Now you can install any go application!

For instance, noti is a nice little utility that triggers an alert (desktop popup, Pushbullet notification, etc.) when a process finishes. From its installation notes, you just run the below to install it:

go get -u github.com/variadico/noti/cmd/noti

Apart from the go applications I suggested here, go out and explore more – go get them 😁

Updating go #

1. Delete the existing $GOROOT directory (not GOPATH!)

   rm -rf ~/go   # as that is my GOROOT
   

2. Download the tar.gz for the latest linux-amd64 binaries.
3. Extract it to the same $GOROOT (~/go in my case).