As I mentioned in the post description above, the solution to clean up the auto-populated ${GOPATH}/pkg/ is to run go clean -modcache. While the solution is simple, it wasn’t easy for me to discover, and so I am writing this short piece to make it a bit more discoverable for others like me.

The $GOPATH directory #

The GOPATH environment variable lists places to look for Go code.

If you haven’t set this variable and if the ${HOME}/go/ directory isn’t used to contain the Go distribution, $GOPATH defaults to that path.

When you run go get to install any Go package,

• ${GOPATH}/pkg/ gets populated with all the Go module dependencies,
• .. and all package’s executables if any get installed to ${GOPATH}/bin/.

To confirm the value of GOPATH used by Go, see go env | rg GOPATH.

The disk space issue #

I have been happily installing and building Go apps like Hugo using go get or go build, and all these installations would end up in the $GOPATH. But over time, I noticed that the disk space used by ${GOPATH}/pkg/ just kept on creeping up.

Today I happened to notice that this directory was taking up roughly 4 GB of my disk space! I started analyzing why it was taking up so much space using my favorite tool for this purpose – ncdu.

Here’s a snapshot showing disk usage by one of the sub-directories under ${GOPATH}/pkg/, which shows the problem — Over time, I had accumulated multiple versions of multiple packages!

Write-protected ${GOPATH}/pkg/ #

I identified the problem. And I knew that I just needed to delete all the old packages.

I had been searching a solution to this on and off, but didn’t have much success, mainly attributed to the short and generic name of the “Go” language, and the fact that I didn’t know what the ${GOPATH}/pkg/ directory was called.

Cleaning up “modcache” #

Today, I finally had success when I searched for this magic phrase: golang “cannot remove” “pkg/mod” .. and Golang Issues # 27161 was the first search result!

The solution was so simple, but so difficult to look for ..

go clean -modcache

From that issue, I also learned that the ${GOPATH}/pkg/ directory is the default “modcache” or the cache directory for holding all the installed Go modules.

Wrapping up #

Knowing that the stuff I was trying to delete is called modcache, this of course works ..

go help clean | rg modcache -A 2

The issue referenced above gets resolved when a -modcacherw switch gets added to the go build command. I see that switch when I run go help build. But that switch is not available for go get? .. because I don’t see it when I run go help get.

I don’t understand why Go decided to make this so complicated by taking away the write access from the user who installed the Go package!

References #

• go Documentation – GOPATH environment variable
• go Documentation – go help clean
• Go Wiki – Setting GOPATH
• Golang Issue # 27161

Today I saw this toot by user @hyperlinkyourheart regarding installing Nim:

Not a great experience so far though - choosenim is broken on Ubuntu 22.04 based systems ..

.. and that inspired this quick post. I would encourage the user posting that to bring up that issue on the Nim Forum, but here’s a quick stop-gap solution to install Nim using just curl and tar.

1. Copy and save the below script somewhere, let’s say as ~/scripts/download_nim.sh. Update the nim_install_dir variable in there to your choice. That is set to ${HOME}/nim by default.

   #!/usr/bin/env bash
   
   # Running this script will download and extract nim installation to ~/nim/${nim_version}.
   # To uninstall nim, just remove the ~/nim directory.
   
   set -euo pipefail # http://redsymbol.net/articles/unofficial-bash-strict-mode
   IFS=$'\n\t'
   
   nim_version="1.6.6"
   nim_archive_url="https://nim-lang.org/download/nim-${nim_version}-linux_x64.tar.xz"
   nim_install_dir="${HOME}/nim"
   
   tmp_dir="/tmp/${USER}"
   nim_download_dir="${tmp_dir}/nim-${nim_version}"
   
   nim_version_dir="${nim_install_dir}/${nim_version}"
   
   mkdir -p "${tmp_dir}"
   cd "${tmp_dir}" || exit
   echo "Downloading nim archive from ${nim_archive_url} .."
   curl -RLs "${nim_archive_url}" -o "nim.tar.xz"
   tar xf nim.tar.xz # Extracts to ${nim_download_dir}.
   
   if [[ -d "${nim_version_dir}" ]]
   then
       rm -rf "${nim_version_dir}"
   fi
   mkdir -p "${nim_version_dir}/doc"
   
   cd "${nim_version_dir}" || exit
   cp -fP "${nim_download_dir}"/doc/*.css ./doc/. # Required for 'nim doc ..' to work
   cp -rfP "${nim_download_dir}"/bin .
   cp -rfP "${nim_download_dir}"/lib .
   cp -rfP "${nim_download_dir}"/compiler . # Required for 'nimterop' package
   cp -rfP "${nim_download_dir}"/config .
   
   cd "${nim_install_dir}" || exit
   find . -name "bin" -type l -delete
   ln -fs "${nim_version_dir}"/bin ./bin
   
   echo "Finished downloading $("${nim_version_dir}"/bin/nim -v | head -n 1)"
   

   Code Snippet 1: Nim download script
2. Make the script an executable: chmod +x ~/scripts/download_nim.sh
3. Run ~/scripts/download_nim.sh.. This will download the nim compiler and its standard libraries (totaling to only 40MB!) to the path set in ${nim_install_dir}.
4. Make sure that the path in ${nim_install_dir} is added to your PATH.
5. Run nim --version to check the installation.

   Nim Compiler Version 1.6.6 [Linux: amd64]
   Compiled at 2022-05-05
   Copyright (c) 2006-2021 by Andreas Rumpf
   
   git hash: 0565a70eab02122ce278b98181c7d1170870865c
   active boot switches: -d:release
   

Now head over to https://nim-lang.org/learn.html to learn this awesome ❤️ language!

I like ❤️ using Nim as my go-to programming language – be it for scripting, some CLI tool or any other application. The nice thing when using Nim is that you don’t have to throw away any of the past work done in C or C++, because it is pretty easy to bind those C/C++ libraries and extend their functionality in Nim.

To demonstrate that extensibility, in this post, I will add some new API in Nim that builds on top of the std::list STL The C++ STL is a set of template classes to provide common programming data structures and functions such as lists, stacks, arrays, etc. library. I won’t be re-writing any of the std::list methods in Nim — No need to re-invent the std::list wheel in Nim! — I will directly use the C++ std::list methods to build new API functions in Nim. 🤯

Goal: Be able to easily access C++ std::list elements #

From the std::list reference, we can see that we cannot access a list element by index using syntax like list_obj[index]. At the end of this post, you will be able to convert C++ list objects to Nim sequences and make the list data super-accessible — You can print them using echo, use built-in libraries like sequtils, and more.

importcpp pragma #

The first thing we’ll need is to be able to call the C++ list methods directly in Nim, and we will need the importcpp pragma This post will highlight some key features of this pragma. For full details, check out this section in the Nim Manual linked in References. for that. This pragma is used to import a C++ method or type into the Nim space.

This pragma is typically used like this:

{.link: "/path/to/libCppCode.so".}

type
  MyNimType {.importcpp: "MyCppType", header: "libCppCode.hpp".} = object

proc myNimProc(): MyNimType {.importcpp: "myCppMethod", header: "libCppCode.hpp".}

var
  foo = MyNimProc()

• MyCppType type from the C++ library is mapped to MyNimType You would typically name the type and method identifiers same same when binding external libraries to Nim, just to prevent any confusion. You can name them differently if you want to though. in the Nim code.
• myCppMethod from C++ is mapped to myNimProc.
• Then the code can call that myNimProc in the Nim code along with other Nim stuff.
• The link pragma is needed if we need to tell the Nim compiler where to find the implementation For the case of linking to the C++ STL <list> header, we do not need to provide the std::list implementation because the GNU GCC compiler provides that. of linked methods.

Bindings to the C++ <list> header #

Below code snippet shows us all the <list> header bindings we will need to meet the goal of this post.

1
2
3
4
5
6
7
8

type
  List*[T]                              {.importcpp: "std::list", header: "<list>".} = object
  ListIter*[T]                          {.importcpp: "std::list<'0>::iterator", header: "<list>".} = object

proc initList*[T](): List[T]            {.importcpp: "std::list<'*0>()", constructor, header: "<list>".}
proc size*(l: List): csize_t            {.importcpp: "size", header: "<list>".}
proc begin*[T](l: List[T]): ListIter[T] {.importcpp: "begin", header: "<list>".}
proc `[]`*[T](it: ListIter[T]): T       {.importcpp: "*#", header: "<list>".}

• Line 2 binds the C++ std::list type with the Nim List type. So, a List[cint] type in Nim get mapped to std::list<int> in C++.
• Line 3 binds the C++ iterator type from std::list with the Nim type ListIter.
  • Note special pattern <'0> See 📖 importcpp for procs for details on such special patterns. in the importcpp argument for this binding. The apostrophe followed by an integer i is replaced by the i’th parameter in the ListIter[T] type i.e. the T type. So, ListIter[cint] in Nim gets mapped to std::list<int>::iterator in C++.
• Line 5 binds the list constructor () with the Nim proc initList[T](). As the constructor returns a list type, the return type on the Nim side set to the equivalent List[T].
  • Note the slightly different pattern in the importcpp argument for this binding: <'*​0>. If we had not used that asterisk there, that would have mapped to the proc’s return type, List[T]. But we want the C++ mapped constructor to look like std::list<int>() i.e. we need T and not List[T] in the C++ template type. The * does the job of deriving T from its “container” type List[T].
• In lines 6 and 7, size and begin on C++ get mapped to same named procs on the Nim side, with the correct return types.
  • csize_t is a Nim type that maps to the C size_t type.
  • As begin returns an iterator handle, the return type of begin is ListIter[T].
  • Note that I did not use the std::list prefix for these mappings, because these procs will always be called in context of their arguments. For example, listIterVar.size() in Nim will map to listIterVar.size() in C++..
• Line 8 maps the C/C++ indirection operator * with Nim dereferencing operator []. The input argument type is an iterator. So if iter is a variable of type ListIter[cint], iter[] will return the value of type cint pointed to by that iterator.
  • The importcpp argument for this mapping has a different pattern: *#. The # is replaced by the first argument of the proc: iter. So iter[] in Nim code iter[] is same as writing `[]`(iter), but one might agree that the former style is better and more readable. will translate to *​iter¹ on the C++ side.

Test C++ code #

We have the <list> bindings ready, but now we need some test C++ code which will be our “library” for use in the Nim code.

So here it is:

void generateList(int num, std::list<int> *retList)
{
    for (int i = 0; i < num; ++i) {
        retList->push_back(i * 2);
    }
}

The C++ “library” here has a generateList API with expects an integer num as the first arg, and it save a num element std::list<int> object to the pointer passed as its second arg.

Now, we can either go the conventional route of compiling this code to a shared object (.so or .dll) and then linking it to the Nim code using the link pragma.

We will do something cooler.. we will inline that C++ code in Nim using the emit pragma and bind to it using importcpp just as we did for the <list> methods! 😎

{.emit: """
void generateList(int num, std::list<int> *retList)
{
    for (int i = 0; i < num; ++i) {
        retList->push_back(i * 2);
    }
}
""".}
proc generateList(num: cint; lPtr: ptr List[cint]) {.importcpp: "generateList(@)".}

Here, I also introduce you readers to a new importcpp pattern: @. The @ expands to comma-separated arguments passed on the Nim side.

Note the equivalence between the C++ function signature void generateList(int num, std::list<int> *​retList) and Nim proc signature proc generateList(num: cint; lPtr: ptr List[cint]).

Test Nim code #

Below Nim code will call the test C++ generateList method defined above and then print the whole list object after converting it to a Nim sequence.

var
  l = initList[cint]()
generateList(10, addr l)
var
  s = l.toSeq()
echo "sequence   = ", s

• In above snippet, initList bindings defined in Code Snippet 2 is used to initialize the list object l.
• Its address or pointer is passed to generateList which will populate the Nim allocated memory with 10 elements.
• Then we call a toSeq proc (that we will define in the next section) to convert that list to a Nim sequence.
• Now that the list object is converted to a Nim sequence, printing it out as easy as you see above 😄.

Extending the std::list API in Nim #

The end goal is to be able to write a toSeq proc that can convert a std::list object to a Nim sequence. Here’s a psuedo-code I began before it evolved into Code Snippet 7.

proc toSeq (l: list_obj): sequence
  lBegin = handle to the iterator for the beginning of list_obj; use `begin`
  lSize = number of elements in list_obj; use `size`

  lIter = lBegin
  for idx in 0 ..< lSize:
    sequence[idx] = *lIter; use the `[]` dereferencing operator
    lIter = pointer to the next element; use `next` from <iterator>

As it turned out, the real Nim code wasn’t longer or more complicated than the pseudocode 😆.

1
2
3
4
5
6
7
8
9

proc next*[T](it: ListIter[T]; n = 1): ListIter[T] {.importcpp: "next(@)", header: "<iterator>".}

proc toSeq*[T](l: List[T]): seq[T] =
  result = newSeq[T](l.size())
  var
    it = l.begin()
  for i in 0 ..< l.size():
    result[i] = it[]
    it = it.next()

• Line 1 - We needed a binding to the std::next method from iterator so that we can increment the iterator starting from the begin() returned value.
• Line 4 - We already know the size of the list object, so we pre-allocated the Nim sequence to the same size by using the newSeq proc.
• Then rest of the code follows the idea presented in the pseudocode above.

Final Code #

The final code to (i) wrap the <list> types and methods and (ii) one method from <iterator>, and (iii) define a proc for converting std::list objects to Nim sequences was only 22 lines! (including comments and blank lines)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

when not defined(cpp): {.error: "This file needs to be compiled with cpp backend: Run 'nim -b:cpp r <file>'.".}

## <list> bindings -- https://en.cppreference.com/w/cpp/container/list
type
  List*[T]                              {.importcpp: "std::list", header: "<list>".} = object
  ListIter*[T]                          {.importcpp: "std::list<'0>::iterator", header: "<list>".} = object

proc initList*[T](): List[T]            {.importcpp: "std::list<'*0>()", constructor, header: "<list>".}
proc size*(l: List): csize_t            {.importcpp: "size", header: "<list>".}
proc begin*[T](l: List[T]): ListIter[T] {.importcpp: "begin", header: "<list>".}
proc `[]`*[T](it: ListIter[T]): T       {.importcpp: "*#", header: "<list>".}

## std::list processing procs and iterators
proc next*[T](it: ListIter[T]; n = 1): ListIter[T] {.importcpp: "next(@)", header: "<iterator>".}

proc toSeq*[T](l: List[T]): seq[T] =
  result = newSeq[T](l.size())
  var
    it = l.begin()
  for i in 0 ..< l.size():
    result[i] = it[]
    it = it.next()

## Test
when isMainModule:
  {.emit: """
  void generateList(int num, std::list<int> *retList)
  {
      for (int i = 0; i < num; ++i) {
          retList->push_back(i * 2);
      }
  }
  """.}
  proc generateList(num: cint; lPtr: ptr List[cint]) {.importcpp: "generateList(@)".}

  var
    l = initList[cint]()
  generateList(10, addr l)
  var
    s = l.toSeq()
  echo "sequence   = ", s
  echo "first elem = ", s[0]
  echo "elem 5     = ", s[5]
  echo "last elem  = ", s[^1]

sequence   = @[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
first elem = 0
elem 5     = 10
last elem  = 18

You can try this code by saving to a list_to_seq.nim file and running nim -b:cpp r list_to_seq.nim.

Alternatively, you can paste this code on https://play.nim-lang.org/, set the ‘Compilation target’ to C++ and hit the yellow ‘Run!’ button.

Summary #

It becomes fairly easy to map any C++ (or C) library to Nim. Once that it done, you can leverage the power of Nim to extend that mapped library or manipulate data generated by that library. And you do not need to be a C++ programmer to do so (I am not).

The folks on Nim Forum, Discord and elsewhere have been very generous with their help and support, and that’s one of the main reasons it’s a joy to use Nim. 🙏

References #

• 📖 Nim Manual – importcpp pragma
• C++ std::list reference
• Nim cppstl This package contains bindings to std::string, std::vector and std::complex as of <2022-03-10 Thu>. package – can be installed using nimble install cppstl

Problem statement #

I’d like to create Nim applications, and then be able to easily deploy the static built binaries so that anyone on GNU/Linux type OS can just download the binary and run them.

If and when I figure out how to do the same for macOS and Windows too, I’ll write a separate post for that. If you can help me out with that, please comment below, or open an issue/PR in the repo linked at the end.

Solution #

The solution is multi-step:

1. Building static binaries (only for GNU/Linux type OS).
2. Optimizing the binary size.
3. Doing the above two easily.
4. Creating and deploying the builds automatcally (on GitHub).

1 Building static binaries #

My primary OS for coding and development is RHEL 6.8.

By default, Nim builds dynamically linked binaries using the glibc version present on the OS where those binaries are built.

Living on a stable but old OS like RHEL 6.8 (that has glibc 2.12 by default), I have ended up with the issue many a times¹ where the deployed binary would be compiled with a newer glibc and so it wouldn’t run on my system. I would need to eventually compile that binary myself.

I don’t want to inflict other people with the same problem.

Inspired by how statically compiled binaries are distributed for tools built in other languages (hugo [Go], ripgrep [Rust], pandoc [Haskell], etc), I wanted the same for whatever I build with Nim.

I kept looking around for a “single press button” flow to do this in Nim for a while, and after finding nothing, I came up a flow myself, that I present in this post.

I picked musl to statically compile my binaries.

What is musl? #

Here’s a one-liner description from Wikipedia:

musl is a C standard library intended for operating systems based on the Linux kernel, released under the MIT License.

Check out its introduction on its official webpage for more details.

Installing musl #

Installing musl was as simple as downloading its .tar.gz and running make && make install with the --prefix configured appropriately. For the examples that follow, let’s say you set the prefix to /usr/local/musl/.

• With the prefix set as above, the musl-gcc binary would get installed in the /usr/local/musl/bin/ directory.
• Add /usr/local/musl/bin/ to your PATH environment variable.

Static linking #

By default, Nim will use the gcc executable that’s found in PATH for C compilation. But using the --gcc.exe switch, we override to use the musl-gcc executable. Similarly, we use the --gcc.linkerexe switch to use musl-gcc too, and then the -static option is passed to the linker using the --passL switch.

With a hello.nim containing:

echo "Hello, World!"

running the below:

# Assuming that musl-gcc binary is found in PATH.
nim --gcc.exe:musl-gcc --gcc.linkerexe:musl-gcc --passL:-static c hello.nim

I got a 98kB statically linked hello binary.

2 Optimizing the binary size #

The binary size can be optimized by Nim itself by doing a Release build (-d:release) which disables runtime checks used for debugging, and enables certain optimizations. Adding --opt:size does further binary-size optimization.

So, with the same hello.nim, running the below:

# Assuming that musl-gcc binary is found in PATH.
nim --gcc.exe:musl-gcc --gcc.linkerexe:musl-gcc --passL:-static -d:release --opt:size c hello.nim

shrunk down the binary size to 43kB.

Upon running external utilities like strip (from binutils)² and upx³, the binary size shrunk even further! – 16kB.

# Assuming that musl-gcc binary is found in PATH.
nim --gcc.exe:musl-gcc --gcc.linkerexe:musl-gcc --passL:-static -d:release --opt:size c hello.nim
strip -s ./hello
upx --best ./hello

3 Doing the above two easily #

All of that is great, but it wouldn’t be fun or elegant to do all the steps in Code Snippet 1 manually (or even in a shell script).

Wouldn’t it be awesome if I could just run a nim musl hello.nim command‽

NimScript #

The “nim musl” command is not available out-of-box, but it is easy to create one using a NimScript config file, which is in the form of a project-specific config.nims.

From the NimScript docs:

Strictly speaking, NimScript is the subset of Nim that can be evaluated by Nim’s builtin virtual machine (VM). This VM is used for Nim’s compiletime function evaluation features, but also replaces Nim’s existing configuration system.

The VM cannot deal with importc, the FFI is not available, so there are not many stdlib modules that you can use with Nim’s VM. However, at least the following modules are available:

• strutils
• math
• distros

That list of modules is not comprehensive. I believe that the macros module should be added to that list too (we also happen to need it for the error macros in the config.nims below 😄).

So, instead of having to resort to the hacky bash scripts, NimScript allows one to write scripts in the awesome Nim syntax, and it’s platform-agnostic too!

Also, as you see below, the tasks in NimScripts basically become user-defined Nim-subcommands. For example, “nim musl” command doesn’t exist, but I could make one just like that by defining a “musl” NimScript task.

config.nims #

Below config.nims is generic and would work for most Nim projects. You need to just put it in the directory from where you compile the Nim code.

Nim devel (v0.18.1) is needed for the below NimScript to work, because the findExe I use in there was not present in Nim v0.18.0.

# config.nims
from macros import error
from ospaths import splitFile, `/`

# -d:musl
when defined(musl):
  var muslGccPath: string
  echo "  [-d:musl] Building a static binary using musl .."
  muslGccPath = findExe("musl-gcc")
  # echo "debug: " & muslGccPath
  if muslGccPath == "":
    error("'musl-gcc' binary was not found in PATH.")
  switch("gcc.exe", muslGccPath)
  switch("gcc.linkerexe", muslGccPath)
  switch("passL", "-static")

proc binOptimize(binFile: string) =
  ## Optimize size of the ``binFile`` binary.
  echo ""
  if findExe("strip") != "":
    echo "Running 'strip -s' .."
    exec "strip -s " & binFile
  if findExe("upx") != "":
    # https://github.com/upx/upx/releases/
    echo "Running 'upx --best' .."
    exec "upx --best " & binFile

# nim musl foo.nim
task musl, "Builds an optimized static binary using musl":
  ## Usage: nim musl <.nim file path>
  let
    numParams = paramCount()
  if numParams != 2:
    error("The 'musl' sub-command needs exactly 1 argument, the Nim file (but " &
      $(numParams-1) & " were detected)." &
      "\n  Usage Example: nim musl FILE.nim.")

  let
    nimFile = paramStr(numParams) ## The nim file name *must* be the last.
    (dirName, baseName, _) = splitFile(nimFile)
    binFile = dirName / baseName  # Save the binary in the same dir as the nim file
    nimArgs = "c -d:musl -d:release --opt:size " & nimFile
  # echo "[debug] nimFile = " & nimFile & ", binFile = " & binFile

  # Build binary
  echo "\nRunning 'nim " & nimArgs & "' .."
  selfExec nimArgs

  # Optimize binary
  binOptimize(binFile)

  echo "\nCreated binary: " & binFile

Here’s a quick breakdown of that script:

• We first import the modules we would need for the sugar-syntax error macro, and for basic file path operations.
• when defined(musl) block is used to do something extra to the nim command if user has passed a -d:musl switch during compilation (this custom compile-time define switch is used in the “musl” task).
• The binOptimize is just a Nim proc to contain the outside-Nim binary size optimization commands.
• Finally, the NimScript task “musl” allows me to define my custom “musl” sub-command for nim so that I can run nim musl FOO.nim.

With a directory containing this config.nims and a FOO.nim, running nim musl FOO.nim will be analogous to manually running the commands in Code Snippet 1.

4 Creating and deploying the builds automatically #

All of that is still great, but ..

And of course, I can do that too! – At least on GitHub using Travis CI.

I would also like to know how to do the same on non-GitHub repos too. If you know, please comment below.

Below is a generic .travis.yml that works for a repo name “FOO” and containing the Nim code in its “src/FOO.nim” file. Notes on what this config file does follow after that snippet.

language: c
sudo: required

cache:
  directories:
    - nim
    - upx

env:
  global:
    - PROGNAME="$(basename ${TRAVIS_BUILD_DIR})" # /travis/build/kaushalmodi/hello_musl -> hello_musl
    - NIMFILE="src/${PROGNAME}.nim"
    - BINFILE="src/${PROGNAME}"
    - ASSETFILE="${PROGNAME}-${TRAVIS_TAG}.Linux_64bit_musl.tar.xz"
    - NIMREPO="https://github.com/nim-lang/Nim"
    - NIMVER="$(git ls-remote ${NIMREPO} devel | cut -f 1)"
    - NIMDIR="${TRAVIS_BUILD_DIR}/nim/${NIMVER}"
    - UPXVER="3.95"             # Change this value when upgrading upx

addons:
  apt:
    packages:
      # For building MUSL static builds on Linux.
      - musl-tools

install:
  - echo "NIMDIR = ${NIMDIR}"
  # After building nim, wipe csources to save on cache space.
  - "{ [ -f ${NIMDIR}/bin/nim ]; } ||
      ( rm -rf nim;
        mkdir -p nim;
        git clone --single-branch --branch devel --depth=1 ${NIMREPO} ${NIMDIR};
        cd ${NIMDIR};
        [ -d csources ] || git clone --depth 1 https://github.com/nim-lang/csources.git;
        cd csources;
        sh build.sh;
        cd ..;
        ./bin/nim c koch;
        ./koch boot -d:release;
        rm -rf csources;
      )"
  - export PATH="${NIMDIR}/bin:${PATH}"
  - nim -v

  - echo "Installing upx .."
  - "{ [ -f upx/${UPXVER}/upx ]; } ||
      { curl -OL https://github.com/upx/upx/releases/download/v${UPXVER}/upx-${UPXVER}-amd64_linux.tar.xz;
        tar xvf upx-${UPXVER}-amd64_linux.tar.xz;
        mkdir -p upx;
        mv upx-${UPXVER}-amd64_linux upx/${UPXVER};
      }"
  - export PATH="${TRAVIS_BUILD_DIR}/upx/${UPXVER}/:${PATH}"
  - upx --version | grep -E '^upx'

script:
  # Ensure that you are in repo/build root now.
  - cd "${TRAVIS_BUILD_DIR}"
  - echo "NIMFILE = ${NIMFILE}"
  - echo "BINFILE = ${BINFILE}"
  # Compile the static binary using musl.
  - nim musl "${NIMFILE}"
  # See that the binary is not dynamic.
  - ldd "${BINFILE}" || true
  # Run the binary.
  - "${BINFILE}"

before_deploy:
  - cd "${TRAVIS_BUILD_DIR}"
  - cp "${BINFILE}" "${PROGNAME}"
  - tar caf "${ASSETFILE}" "${PROGNAME}"
deploy:
  provider: releases
  api_key: "${GITHUB_OAUTH_TOKEN}"
  file: "${ASSETFILE}"
  skip_cleanup: true
  on:
    tags: true

• I like environment variables to do all the configuration. So I do that in the env / global block.
• Caching is enabled for “nim” and “upx” directories, which are created in the install block.
  • So if the Nim devel branch hasn’t been updated since the last Travis build, the cached Nim build is used.
  • And similar applies to the upx binary, which relies on the value of the UPXVER env variable.
• In the install block, nim is built from the Nim devel branch HEAD, and upx binary is downloaded from the Releases section of its repo.
• The key command is nim musl "${NIMFILE}" in the script block, which uses the musl NimScript task defined in the config.nims in the same repo.
• In before_deploy, tar is used to create an archive of the generated binary.
• The deploy section is mainly copy-paste of what’s suggested in the Travis CI documentation.
  • About the ${GITHUB_OAUTH_TOKEN}, the first step is to get your GitHub account’s repo level token from GitHub Tokens settings.
  • The second step is to create a custom environment variable named GITHUB_OAUTH_TOKEN in that repo’s Travis CI settings (ensure that the variable’s value is not made public in the Travis CI’s logs, though they are private by default), and assigning that token string to that variable.

How it works in practice #

Let’s assume that the one-time setup for a new repo is already done. The config.nims and .travis.yml are committed and pushed to the GitHub repo, and the GITHUB_OAUTH_TOKEN environment variable is set in the repo’s Travis settings.

Now, to auto-generate the assets for each release, all I need to do is:

1. Commit my changes to the Nim code.
2. Tag the commit.
3. Push the commit and tag to GitHub.

Repo #

You can find the latest versions of the above code in my Nim+musl template repo hello_musl.

You can also try downloading and running (on a GNU/Linux 64-bit OS) the deployed static binary from its Releases section and see this printed in full glory! 😄

Hello, World!

References #

• r/nim – Static Linking with Nim
• hookrace.net – Statically linking against musl libc
• Using NimScript for configuration
• NimScript “API”

Today I came across this FizzBuzz attempt for Nim, so I thought of giving it a try too.

Here’s how Fizz buzz is defined on Wikipedia:

Fizz buzz is a group word game for children to teach them about division. Players take turns to count incrementally, replacing any number divisible by three with the word “fizz”, and any number divisible by five with the word “buzz”.

And it’s also one of those basic programming problems (labeled as interview questions) that’s written in many languages. Here’s how Rosetta Code defines this programming task —

Write a program that prints the integers from 1 to 100 (inclusive).

But:

• for multiples of three, print Fizz (instead of the number)
• for multiples of five, print Buzz (instead of the number)
• for multiples of both three and five, print FizzBuzz (instead of the number)

I am not doing anything radical in this post.. just recording my attempt at FizzBuzz using Nim 😄.

for i in 1 .. 100:
  var str = $i
  if (i mod 3) == 0:
    str = "Fizz"
    if (i mod 5) == 0:
      str.add("Buzz")
  elif (i mod 5) == 0:
    str = "Buzz"
  echo str

1
2
Fizz
4
Buzz
Fizz
7
8
Fizz
Buzz
11
Fizz
13
14
FizzBuzz
16
17
Fizz
19
Buzz
Fizz
22
23
Fizz
Buzz
26
Fizz
28
29
FizzBuzz
31
32
Fizz
34
Buzz
Fizz
37
38
Fizz
Buzz
41
Fizz
43
44
FizzBuzz
46
47
Fizz
49
Buzz
Fizz
52
53
Fizz
Buzz
56
Fizz
58
59
FizzBuzz
61
62
Fizz
64
Buzz
Fizz
67
68
Fizz
Buzz
71
Fizz
73
74
FizzBuzz
76
77
Fizz
79
Buzz
Fizz
82
83
Fizz
Buzz
86
Fizz
88
89
FizzBuzz
91
92
Fizz
94
Buzz
Fizz
97
98
Fizz
Buzz

When writing bash scripts, I often need to know if the script is receiving input from the terminal, or some piped process. I would also need to know if the script is sending output to the terminal, or to another piped process.

As I am learning Nim and trying to write new scripts using that, I need to know how to do the same in Nim.

Today, I came across this r/nim thread where a user needed to know if the Nim-compiled binary was receiving input from stdin.

That reminded me of this technique that I had used in my bash project eless.. (by checking [[ -t 0 ]] and [[ -t 1 ]]).

Bash #

Here’s a similar bash snippet if you want to try it out:

#!/usr/bin/env bash

# How to detect whether input is from keyboard, a file, or another process.
# Useful for writing a script that can read from standard input, or prompt the
# user for input if there is none.

# https://gist.github.com/davejamesmiller/1966557
if [[ -t 0 ]] # Script is called normally - Terminal input (keyboard) - interactive
then
    # eless foo
    # eless foo | cat -
    echo "--> Input from terminal"
else # Script is getting input from pipe or file - non-interactive
    # echo bar | eless foo
    # echo bar | eless foo | cat -
    echo "--> Input from PIPE/FILE"
fi

# https://stackoverflow.com/a/911213/1219634
if [[ -t 1 ]] # Output is going to the terminal
then
    # eless foo
    # echo bar | eless foo
    echo "    Output to terminal -->"
else # Output is going to a pipe, file?
    # eless foo | cat -
    # echo bar | eless foo | cat -
    echo "    Output to a PIPE -->"
fi

So using that as the basis, I went down the path of figuring out how to do the same in Nim.

Nim #

I remembered seeing the variables stdin and stdout defined in the implicitly¹ imported system module. So I started with trying to get the value of those variables and see if they helped me figure out how they are associated with the terminal.

Using os.getFileInfo #

From the docs of stdin and stdout, I learned that the type of both of those is File. So while searching for “File”, I ended up on os.FileInfo, whose value I can get using os.getFileInfo.

Based on the structure of that FileInfo object:

FileInfo = object
  id*: tuple[device: DeviceId, file: FileId]
  kind*: PathComponent
  size*: BiggestInt
  permissions*: set[FilePermission]
  linkCount*: BiggestInt
  lastAccessTime*: times.Time
  lastWriteTime*: times.Time
  creationTime*: times.Time

I guessed that the id field in there might hold the information I needed.. and indeed it did!

By printing out the value of getFileInfo(stdin).id.file, and running the Nim-compiled binary by itself (./binary) vs providing it the output of another process (echo foo | ./binary), I learned that its value is 35 if the binary is not receiving input from another process/pipe.

Similarly the value of getFileInfo(stdout).id.file is also 35 if the binary is not sending the output to another process/pipe.

Based on that deduction, this worked!

# Figuring out if input is coming from a pipe and if output is going to a pipe.
import std/[os, strutils]

# https://nim-lang.org/docs/os.html#FileInfo
if getFileInfo(stdin).id.file==35:
  # ./stdin_stdout foo
  # ./stdin_stdout foo | cat
  echo "--> Input from terminal"
else:
  # echo bar | ./stdin_stdout
  # echo bar | ./stdin_stdout | cat
  echo "--> Input from a PIPE/FILE: `" & readAll(stdin).strip() & "'"

if getFileInfo(stdout).id.file==35:
  # ./stdin_stdout foo
  # echo bar | ./stdin_stdout foo
  echo "    Output to terminal -->"
else:
  # ./stdin_stdout | cat
  # echo bar | ./stdin_stdout | cat
  echo "    Output to a PIPE -->"

Using terminal.isatty #

In the same reddit thread, /u/bpbio from Reddit provides a better, concise answer—using the isatty proc from the terminal module:

proc isatty(f: File): bool {.raises: [], tags: [].}

Returns true if f is associated with a terminal device.

From my brief testing, I saw that isatty(stdin) is equivalent to getFileInfo(stdin).id.file==35, and the same for stdout too.

So my above snippet can be rewritten as:

# Figuring out if input is coming from a pipe and if output is going to a pipe.
import std/[terminal, strutils]

if isatty(stdin):
  # ./stdin_stdout foo
  # ./stdin_stdout foo | cat
  echo "--> Input from terminal"
else:
  # echo bar | ./stdin_stdout
  # echo bar | ./stdin_stdout | cat
  echo "--> Input from a PIPE/FILE: `" & readAll(stdin).strip() & "'"

if isatty(stdout):
  # ./stdin_stdout foo
  # echo bar | ./stdin_stdout foo
  echo "    Output to terminal -->"
else:
  # ./stdin_stdout | cat
  # echo bar | ./stdin_stdout | cat
  echo "    Output to a PIPE -->"

Result #

Assuming that the Nim-compiled binary of the above code² is stdin_stdout, you get this output:

> ./stdin_stdout
--> Input from terminal
    Output to terminal -->
> ./stdin_stdout | cat
--> Input from terminal
    Output to a PIPE -->
> echo foo | ./stdin_stdout
--> Input from a PIPE/FILE: `foo'
    Output to terminal -->
> echo foo | ./stdin_stdout | cat
--> Input from a PIPE/FILE: `foo'
    Output to a PIPE -->