static on A Scripter's Notes

Generics (not exactly) in SystemVerilog

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Fri, 11 Feb 2022 01:48:00 -0500

Using parameterized classes with static functions to make up for the lack of generics in SystemVerilog.

Table of Contents

A real Generics example
Poor man’s Generics in SystemVerilog
Final Code
Cluelib

Many modern languages like Nim allow defining Generic functions types, etc that can work for any type. See Nim Manual 2022 Generics.

This post demonstrates how you can make generics kind of (if you really squint your eyes, you can see it 😄) work with SystemVerilog.

A real Generics example #

The focus of this post is not how to write generics in Nim, but here’s a quick summary:

Line 1 overloads the + operator for any type T.
Line 2 is a compile time check (using when) to see if T is a string, and then concatenates a and b inputs using the & string concat operator.
For other types, it does the expected “a + b”.

1
2
3
4
5
6
7
8
9


proc `+`[T](a, b: T): T =
 when T is string:
 a & b
 else:
 a + b

echo 1 + 2
echo 100.1 + 100.2
echo "x" + "y" + "z"

Code Snippet 1: Example of generics in Nim

3
200.3
xyz

We will now see how to write something like this in SystemVerilog.

Poor man’s Generics in SystemVerilog #

SystemVerilog is a strongly typed compiled language. So you need to define the types for function and task arguments.

So if you need an add function for integers, you would do:

function int add_int(input int a, b);
 return a + b;
endfunction

Code Snippet 2: add_int for adding integers

If you need an add function for real numbers, you would do:

function real add_real(input real a, b);
 return a + b;
endfunction

Code Snippet 3: add_real for adding reals (doubles)

I am using the add functions here only to demonstrate the concept. In real world code, you would come across many cases where you would want to avoid such function redefinitions. Some examples are: dealing with objects of different classes, or queues with elements of different types (e.g. queues of ints, queues of strings), or dynamic arrays of different types, etc.

Parameterized Classes #

In add_int, we see that the type int has to be specified in the function signature. That int can be replaced by a parameter representing a type only if that function is defined inside a parameterized class (See § Parameterized classes 2018, sec. 8.25).

So it would look like this:

class math #(parameter type T = int);
 function T add(input T a, b);
 return a + b;
 endfunction
endclass

Code Snippet 4: Parameterized class and method

This code, though has a problem — In order to call that function, we would first need to construct an object of that class for each type T we plan to use.

Static methods #

The solution to that is to declare the methods/functions in the class as static. That way, that method can be called directly without constructing the object first. See § Static methods 2018, sec. 8.10.

And because we don’t need to construct an object of that class, and don’t want anyone else to unnecessary construct it, we declare the class as virtual.

virtual class math #(parameter type T = int);
 static function T add(input T a, b);
 return a + b;
 endfunction
endclass

Code Snippet 5: Virtual parameterized class and static method

Now we can call math #(int)::add(1, 2) and math #(real)::add(100.1, 100.2) and get the expected results.

Should compile with all types T #

But.. what if the type T is string ..

$typename system function (See § Type name function 2018, sec. 20.6.1) comes to our help here. This function takes in a variable identifier and returns a string name of that variable’s type. So if a variable x is a string, $typename(x) will return "string".

So, we can define the add method like this, right?

virtual class math #(parameter type T = int);
 static function T add(input T a, b);
 if ($typename(a) == "string")
 return {a, b};
 else
 return a + b;
 endfunction
endclass

Code Snippet 6: add method with a condition for string type – Won't compile!

❌ Wrong ❌

Even if you have the code execute conditionally based on the $typename, note that the if ($typename(a) == "string") check is happening at run time, and so the entire code needs to compile for any type T that’s planned to be used.

In the above example, because the {a, b} concatenation isn’t meant for types like ints and reals, it will fail compilation. That’s the reason for the “(not exactly)” in this post’s title 😄

In order to make add work with strings as well, we need this messy cast when the type name is “string” : $cast(ret_val, $sformatf("%s%s", a, b));. It’s basically telling the compiler: “Trust me, I know what I am doing.. I will be executing this code only when the ret_val is a string.”

Final Code #

Finally, here’s the entire code with the generic add method and some test code.

virtual class math #(parameter type T = int);

 static function T add(input T a, b);
 if ($typename(a) == "string") begin
 T ret_val;
 $cast(ret_val, $sformatf("%s%s", a, b));
 return ret_val;
 end else begin
 return a + b;
 end
 endfunction : add

endclass : math

module test;

 $display("1 + 2 = %p", math #(int)::add(1, 2)); // 3
 $display("1.1 + 2.2 = %p", math #(real)::add(1.1, 2.2)); // 3.3
 $display("1 + 1 = %p", math #(bit)::add(1, 1)); // 0
 $display("abc + def = %p", math #(string)::add("abc", "def")); // "abcdef"

 $finish;
 end
endmodule : test

Code Snippet 7: Parameterized class math with static function add

Cluelib #

If you liked how these “generics” work in SystemVerilog and how the math #(int)::add(1, 2)); syntax looks, check out the cluelib library by cluelogic.com. It is an amazing SystemVerilog library with a big collection of “generic” functions like above for handling strings, queues and dynamic arrays, and a lot more. You can take a peek at its API here.

References #

IEEE Standard for SystemVerilog–Unified Hardware Design, Specification, and Verification Language. (2018). IEEE Std 1800-2017 (Revision of IEEE Std 1800-2012), 1–1315. https://doi.org/10.1109/IEEESTD.2018.8299595

Nim contributors. (2022). Nim Manual [Website]. In Nim. https://nim-lang.org/docs/manual.html

Nim: Deploying static binaries

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Mon, 24 Sep 2018 01:47:00 -0400

Deploying Nim applications as static binaries for GNU/Linux type operating systems, built using musl.

Table of Contents

Problem statement
Solution
1 Building static binaries
2 Optimizing the binary size
3 Doing the above two easily
- NimScript
- config.nims
4 Creating and deploying the builds automatically
How it works in practice
Repo
References

Problem statement #

I’d like to create Nim applications, and then be able to easily deploy the static built binaries so that anyone on GNU/Linux type OS can just download the binary and run them.

If and when I figure out how to do the same for macOS and Windows too, I’ll write a separate post for that. If you can help me out with that, please comment below, or open an issue/PR in the repo linked at the end.

Solution #

The solution is multi-step:

Building static binaries (only for GNU/Linux type OS).
Optimizing the binary size.
Doing the above two easily.
Creating and deploying the builds automatcally (on GitHub).

1 Building static binaries #

My primary OS for coding and development is RHEL 6.8.

By default, Nim builds dynamically linked binaries using the glibc version present on the OS where those binaries are built.

Living on a stable but old OS like RHEL 6.8 (that has glibc 2.12 by default), I have ended up with the issue many a times¹ where the deployed binary would be compiled with a newer glibc and so it wouldn’t run on my system. I would need to eventually compile that binary myself.

I don’t want to inflict other people with the same problem.

Inspired by how statically compiled binaries are distributed for tools built in other languages (hugo [Go], ripgrep [Rust], pandoc [Haskell], etc), I wanted the same for whatever I build with Nim.

I kept looking around for a “single press button” flow to do this in Nim for a while, and after finding nothing, I came up a flow myself, that I present in this post.

I picked musl to statically compile my binaries.

What is musl? #

Here’s a one-liner description from Wikipedia:

musl is a C standard library intended for operating systems based on the Linux kernel, released under the MIT License.

Check out its introduction on its official webpage for more details.

Installing musl #

Installing musl was as simple as downloading its .tar.gz and running make && make install with the --prefix configured appropriately. For the examples that follow, let’s say you set the prefix to /usr/local/musl/.

Do not install musl at the default prefix location, else it will override a lot of default header files – an outcome that you probably don’t want.

With the prefix set as above, the musl-gcc binary would get installed in the /usr/local/musl/bin/ directory.
Add /usr/local/musl/bin/ to your PATH environment variable.

Static linking #

By default, Nim will use the gcc executable that’s found in PATH for C compilation. But using the --gcc.exe switch, we override to use the musl-gcc executable. Similarly, we use the --gcc.linkerexe switch to use musl-gcc too, and then the -static option is passed to the linker using the --passL switch.

With a hello.nim containing:

echo "Hello, World!"

running the below:

# Assuming that musl-gcc binary is found in PATH.
nim --gcc.exe:musl-gcc --gcc.linkerexe:musl-gcc --passL:-static c hello.nim

I got a 98kB statically linked hello binary.

2 Optimizing the binary size #

The binary size can be optimized by Nim itself by doing a Release build (-d:release) which disables runtime checks used for debugging, and enables certain optimizations. Adding --opt:size does further binary-size optimization.

So, with the same hello.nim, running the below:

# Assuming that musl-gcc binary is found in PATH.
nim --gcc.exe:musl-gcc --gcc.linkerexe:musl-gcc --passL:-static -d:release --opt:size c hello.nim

shrunk down the binary size to 43kB.

Upon running external utilities like strip (from binutils)² and upx³, the binary size shrunk even further! – 16kB.

# Assuming that musl-gcc binary is found in PATH.
nim --gcc.exe:musl-gcc --gcc.linkerexe:musl-gcc --passL:-static -d:release --opt:size c hello.nim
strip -s ./hello
upx --best ./hello

Code Snippet 1: Static linking using musl-gcc plus binary size optimization

3 Doing the above two easily #

All of that is great, but it wouldn’t be fun or elegant to do all the steps in Code Snippet 1 manually (or even in a shell script).

Wouldn’t it be awesome if I could just run a nim musl hello.nim command‽

— And that’s what I do! 😎

NimScript #

The “nim musl” command is not available out-of-box, but it is easy to create one using a NimScript config file, which is in the form of a project-specific config.nims.

Think of NimScripts as much superior “bash scripts” as they are written in Nim 😄

From the NimScript docs:

Strictly speaking, NimScript is the subset of Nim that can be evaluated by Nim’s builtin virtual machine (VM). This VM is used for Nim’s compiletime function evaluation features, but also replaces Nim’s existing configuration system.

The VM cannot deal with importc, the FFI is not available, so there are not many stdlib modules that you can use with Nim’s VM. However, at least the following modules are available:

strutils

math

distros

That list of modules is not comprehensive. I believe that the macros module should be added to that list too (we also happen to need it for the error macros in the config.nims below 😄).

So, instead of having to resort to the hacky bash scripts, NimScript allows one to write scripts in the awesome Nim syntax, and it’s platform-agnostic too!

Also, as you see below, the tasks in NimScripts basically become user-defined Nim-subcommands. For example, “nim musl” command doesn’t exist, but I could make one just like that by defining a “musl” NimScript task.

`config.nims` #

Below config.nims is generic and would work for most Nim projects. You need to just put it in the directory from where you compile the Nim code.

Nim devel (v0.18.1) is needed for the below NimScript to work, because the findExe I use in there was not present in Nim v0.18.0.

# config.nims
from macros import error
from ospaths import splitFile, `/`

# -d:musl
when defined(musl):
 var muslGccPath: string
 echo " [-d:musl] Building a static binary using musl .."
 muslGccPath = findExe("musl-gcc")
 # echo "debug: " & muslGccPath
 if muslGccPath == "":
 error("'musl-gcc' binary was not found in PATH.")
 switch("gcc.exe", muslGccPath)
 switch("gcc.linkerexe", muslGccPath)
 switch("passL", "-static")

proc binOptimize(binFile: string) =
 ## Optimize size of the ``binFile`` binary.
 echo ""
 if findExe("strip") != "":
 echo "Running 'strip -s' .."
 exec "strip -s " & binFile
 if findExe("upx") != "":
 # https://github.com/upx/upx/releases/
 echo "Running 'upx --best' .."
 exec "upx --best " & binFile

# nim musl foo.nim
task musl, "Builds an optimized static binary using musl":
 ## Usage: nim musl <.nim file path>
 let
 numParams = paramCount()
 if numParams != 2:
 error("The 'musl' sub-command needs exactly 1 argument, the Nim file (but " &
 $(numParams-1) & " were detected)." &
 "\n Usage Example: nim musl FILE.nim.")

 let
 nimFile = paramStr(numParams) ## The nim file name *must* be the last.
 (dirName, baseName, _) = splitFile(nimFile)
 binFile = dirName / baseName # Save the binary in the same dir as the nim file
 nimArgs = "c -d:musl -d:release --opt:size " & nimFile
 # echo "[debug] nimFile = " & nimFile & ", binFile = " & binFile

 # Build binary
 echo "\nRunning 'nim " & nimArgs & "' .."
 selfExec nimArgs

 # Optimize binary
 binOptimize(binFile)

 echo "\nCreated binary: " & binFile

Code Snippet 2: config.nims defining the musl NimScript task

Here’s a quick breakdown of that script:

We first import the modules we would need for the sugar-syntax error macro, and for basic file path operations.
when defined(musl) block is used to do something extra to the nim command if user has passed a -d:musl switch during compilation (this custom compile-time define switch is used in the “musl” task).
The binOptimize is just a Nim proc to contain the outside-Nim binary size optimization commands.
Finally, the NimScript task “musl” allows me to define my custom “musl” sub-command for nim so that I can run nim musl FOO.nim.

With a directory containing this config.nims and a FOO.nim, running nim musl FOO.nim will be analogous to manually running the commands in Code Snippet 1.

4 Creating and deploying the builds automatically #

All of that is still great, but ..

Wouldn’t it be awesome if I could do all of that automatically to create static binary releases for my Nim projects‽

And of course, I can do that too! – At least on GitHub using Travis CI.

I would also like to know how to do the same on non-GitHub repos too. If you know, please comment below.

Below is a generic .travis.yml that works for a repo name “FOO” and containing the Nim code in its “src/FOO.nim” file. Notes on what this config file does follow after that snippet.

language: c
sudo: required

cache:
 directories:
 - nim
 - upx

env:
 global:
 - PROGNAME="$(basename ${TRAVIS_BUILD_DIR})" # /travis/build/kaushalmodi/hello_musl -> hello_musl
 - NIMFILE="src/${PROGNAME}.nim"
 - BINFILE="src/${PROGNAME}"
 - ASSETFILE="${PROGNAME}-${TRAVIS_TAG}.Linux_64bit_musl.tar.xz"
 - NIMREPO="https://github.com/nim-lang/Nim"
 - NIMVER="$(git ls-remote ${NIMREPO} devel | cut -f 1)"
 - NIMDIR="${TRAVIS_BUILD_DIR}/nim/${NIMVER}"
 - UPXVER="3.95"  # Change this value when upgrading upx

addons:
 apt:
 packages:
 # For building MUSL static builds on Linux.
 - musl-tools

install:
 - echo "NIMDIR = ${NIMDIR}"
 # After building nim, wipe csources to save on cache space.
 - "{ [ -f ${NIMDIR}/bin/nim ]; } ||
 ( rm -rf nim;
 mkdir -p nim;
 git clone --single-branch --branch devel --depth=1 ${NIMREPO} ${NIMDIR};
 cd ${NIMDIR};
 [ -d csources ] || git clone --depth 1 https://github.com/nim-lang/csources.git;
 cd csources;
 sh build.sh;
 cd ..;
 ./bin/nim c koch;
 ./koch boot -d:release;
 rm -rf csources;
 )"
 - export PATH="${NIMDIR}/bin:${PATH}"
 - nim -v

 - echo "Installing upx .."
 - "{ [ -f upx/${UPXVER}/upx ]; } ||
 { curl -OL https://github.com/upx/upx/releases/download/v${UPXVER}/upx-${UPXVER}-amd64_linux.tar.xz;
 tar xvf upx-${UPXVER}-amd64_linux.tar.xz;
 mkdir -p upx;
 mv upx-${UPXVER}-amd64_linux upx/${UPXVER};
 }"
 - export PATH="${TRAVIS_BUILD_DIR}/upx/${UPXVER}/:${PATH}"
 - upx --version | grep -E '^upx'

script:
 # Ensure that you are in repo/build root now.
 - cd "${TRAVIS_BUILD_DIR}"
 - echo "NIMFILE = ${NIMFILE}"
 - echo "BINFILE = ${BINFILE}"
 # Compile the static binary using musl.
 - nim musl "${NIMFILE}"
 # See that the binary is not dynamic.
 - ldd "${BINFILE}" || true
 # Run the binary.
 - "${BINFILE}"

before_deploy:
 - cd "${TRAVIS_BUILD_DIR}"
 - cp "${BINFILE}" "${PROGNAME}"
 - tar caf "${ASSETFILE}" "${PROGNAME}"
deploy:
 provider: releases
 api_key: "${GITHUB_OAUTH_TOKEN}"
 file: "${ASSETFILE}"
 skip_cleanup: true
 on:
 tags: true

Code Snippet 3: GitHub Travis CI config file to generate musl-built Release assets

I like environment variables to do all the configuration. So I do that in the env / global block.
Caching is enabled for “nim” and “upx” directories, which are created in the install block.
- So if the Nim devel branch hasn’t been updated since the last Travis build, the cached Nim build is used.
- And similar applies to the upx binary, which relies on the value of the UPXVER env variable.
In the install block, nim is built from the Nim devel branch HEAD, and upx binary is downloaded from the Releases section of its repo.
The key command is nim musl "${NIMFILE}" in the script block, which uses the musl NimScript task defined in the config.nims in the same repo.
In before_deploy, tar is used to create an archive of the generated binary.
The deploy section is mainly copy-paste of what’s suggested in the Travis CI documentation.
- About the ${GITHUB_OAUTH_TOKEN}, the first step is to get your GitHub account’s repo level token from GitHub Tokens settings.
- The second step is to create a custom environment variable named GITHUB_OAUTH_TOKEN in that repo’s Travis CI settings (ensure that the variable’s value is not made public in the Travis CI’s logs, though they are private by default), and assigning that token string to that variable.

How it works in practice #

Let’s assume that the one-time setup for a new repo is already done. The config.nims and .travis.yml are committed and pushed to the GitHub repo, and the GITHUB_OAUTH_TOKEN environment variable is set in the repo’s Travis settings.

Now, to auto-generate the assets for each release, all I need to do is:

Commit my changes to the Nim code.
Tag the commit.
Push the commit and tag to GitHub.

Repo #

You can find the latest versions of the above code in my Nim+musl template repo hello_musl.

You can also try downloading and running (on a GNU/Linux 64-bit OS) the deployed static binary from its Releases section and see this printed in full glory! 😄

Hello, World!

References #

One example ↩︎
The binutils strip utility is used to remove complete symbol tables and other debug info that are not needed for running the executable. ↩︎
From its docs, UPX is an advanced executable file compressor. UPX will typically reduce the file size of programs and DLLs by around 50%-70%, thus reducing disk space, network load times, download times and other distribution and storage costs. ↩︎

static on A Scripter's Notes

Generics (not exactly) in SystemVerilog

A real Generics example #

Poor man’s Generics in SystemVerilog #

Parameterized Classes #

Static methods #

Should compile with all types T #

Final Code #

Cluelib #

References #

Nim: Deploying static binaries

Problem statement #

Solution #

1 Building static binaries #

What is musl? #

Installing musl #

Static linking #

2 Optimizing the binary size #

3 Doing the above two easily #

NimScript #

config.nims #

4 Creating and deploying the builds automatically #

How it works in practice #

Repo #

References #

`config.nims` #