Binding Nim to C++ std::list

I like ❤️ using Nim as my go-to programming language – be it for scripting, some CLI tool or any other application. The nice thing when using Nim is that you don’t have to throw away any of the past work done in C or C++, because it is pretty easy to bind those C/C++ libraries and extend their functionality in Nim.

To demonstrate that extensibility, in this post, I will add some new API in Nim that builds on top of the std::list STL  The C++ STL is a set of template classes to provide common programming data structures and functions such as lists, stacks, arrays, etc. library. I won’t be re-writing any of the std::list methods in Nim — No need to re-invent the std::list wheel in Nim! — I will directly use the C++ std::list methods to build new API functions in Nim. 🤯

Goal: Be able to easily access C++ std::list elements #

From the std::list reference, we can see that we cannot access a list element by index using syntax like list_obj[index]. At the end of this post, you will be able to convert C++ list objects to Nim sequences and make the list data super-accessible — You can print them using echo, use built-in libraries like sequtils, and more.

importcpp pragma #

The first thing we’ll need is to be able to call the C++ list methods directly in Nim, and we will need the importcpp pragma  This post will highlight some key features of this pragma. For full details, check out this section in the Nim Manual linked in References. for that. This pragma is used to import a C++ method or type into the Nim space.

This pragma is typically used like this:

{.link: "/path/to/libCppCode.so".}

type
  MyNimType {.importcpp: "MyCppType", header: "libCppCode.hpp".} = object

proc myNimProc(): MyNimType {.importcpp: "myCppMethod", header: "libCppCode.hpp".}

var
  foo = MyNimProc()

• MyCppType type from the C++ library is mapped to MyNimType  You would typically name the type and method identifiers same same when binding external libraries to Nim, just to prevent any confusion. You can name them differently if you want to though. in the Nim code.
• myCppMethod from C++ is mapped to myNimProc.
• Then the code can call that myNimProc in the Nim code along with other Nim stuff.
• The link pragma is needed if we need to tell the Nim compiler where to find the implementation  For the case of linking to the C++ STL <list> header, we do not need to provide the std::list implementation because the GNU GCC compiler provides that. of linked methods.

Bindings to the C++ <list> header #

Below code snippet shows us all the <list> header bindings we will need to meet the goal of this post.

1
2
3
4
5
6
7
8

type
  List*[T]                              {.importcpp: "std::list", header: "<list>".} = object
  ListIter*[T]                          {.importcpp: "std::list<'0>::iterator", header: "<list>".} = object

proc initList*[T](): List[T]            {.importcpp: "std::list<'*0>()", constructor, header: "<list>".}
proc size*(l: List): csize_t            {.importcpp: "size", header: "<list>".}
proc begin*[T](l: List[T]): ListIter[T] {.importcpp: "begin", header: "<list>".}
proc `[]`*[T](it: ListIter[T]): T       {.importcpp: "*#", header: "<list>".}

• Line 2 binds the C++ std::list type with the Nim List type. So, a List[cint] type in Nim get mapped to std::list<int> in C++.
• Line 3 binds the C++ iterator type from std::list with the Nim type ListIter.
  • Note special pattern <'0>  See 📖 importcpp for procs for details on such special patterns. in the importcpp argument for this binding. The apostrophe followed by an integer i is replaced by the i’th parameter in the ListIter[T] type i.e. the T type. So, ListIter[cint] in Nim gets mapped to std::list<int>::iterator in C++.
• Line 5 binds the list constructor () with the Nim proc initList[T](). As the constructor returns a list type, the return type on the Nim side set to the equivalent List[T].
  • Note the slightly different pattern in the importcpp argument for this binding: <'*​0>. If we had not used that asterisk there, that would have mapped to the proc’s return type, List[T]. But we want the C++ mapped constructor to look like std::list<int>() i.e. we need T and not List[T] in the C++ template type. The * does the job of deriving T from its “container” type List[T].
• In lines 6 and 7, size and begin on C++ get mapped to same named procs on the Nim side, with the correct return types.
  • csize_t is a Nim type that maps to the C size_t type.
  • As begin returns an iterator handle, the return type of begin is ListIter[T].
  • Note that I did not use the std::list prefix for these mappings, because these procs will always be called in context of their arguments. For example, listIterVar.size() in Nim will map to listIterVar.size() in C++..
• Line 8 maps the C/C++ indirection operator * with Nim dereferencing operator []. The input argument type is an iterator. So if iter is a variable of type ListIter[cint], iter[] will return the value of type cint pointed to by that iterator.
  • The importcpp argument for this mapping has a different pattern: *#. The # is replaced by the first argument of the proc: iter. So iter[] in Nim code  iter[] is same as writing `[]`(iter), but one might agree that the former style is better and more readable. will translate to *​iter¹ on the C++ side.

Test C++ code #

We have the <list> bindings ready, but now we need some test C++ code which will be our “library” for use in the Nim code.

So here it is:

void generateList(int num, std::list<int> *retList)
{
    for (int i = 0; i < num; ++i) {
        retList->push_back(i * 2);
    }
}

The C++ “library” here has a generateList API with expects an integer num as the first arg, and it save a num element std::list<int> object to the pointer passed as its second arg.

Now, we can either go the conventional route of compiling this code to a shared object (.so or .dll) and then linking it to the Nim code using the link pragma.

We will do something cooler.. we will inline that C++ code in Nim using the emit pragma and bind to it using importcpp just as we did for the <list> methods! 😎

{.emit: """
void generateList(int num, std::list<int> *retList)
{
    for (int i = 0; i < num; ++i) {
        retList->push_back(i * 2);
    }
}
""".}
proc generateList(num: cint; lPtr: ptr List[cint]) {.importcpp: "generateList(@)".}

Here, I also introduce you readers to a new importcpp pattern: @. The @ expands to comma-separated arguments passed on the Nim side.

Note the equivalence between the C++ function signature void generateList(int num, std::list<int> *​retList) and Nim proc signature proc generateList(num: cint; lPtr: ptr List[cint]).

Test Nim code #

Below Nim code will call the test C++ generateList method defined above and then print the whole list object after converting it to a Nim sequence.

var
  l = initList[cint]()
generateList(10, addr l)
var
  s = l.toSeq()
echo "sequence   = ", s

• In above snippet, initList bindings defined in Code Snippet 2 is used to initialize the list object l.
• Its address or pointer is passed to generateList which will populate the Nim allocated memory with 10 elements.
• Then we call a toSeq proc (that we will define in the next section) to convert that list to a Nim sequence.
• Now that the list object is converted to a Nim sequence, printing it out as easy as you see above 😄.

Extending the std::list API in Nim #

The end goal is to be able to write a toSeq proc that can convert a std::list object to a Nim sequence. Here’s a psuedo-code I began before it evolved into Code Snippet 7.

proc toSeq (l: list_obj): sequence
  lBegin = handle to the iterator for the beginning of list_obj; use `begin`
  lSize = number of elements in list_obj; use `size`

  lIter = lBegin
  for idx in 0 ..< lSize:
    sequence[idx] = *lIter; use the `[]` dereferencing operator
    lIter = pointer to the next element; use `next` from <iterator>

As it turned out, the real Nim code wasn’t longer or more complicated than the pseudocode 😆.

1
2
3
4
5
6
7
8
9

proc next*[T](it: ListIter[T]; n = 1): ListIter[T] {.importcpp: "next(@)", header: "<iterator>".}

proc toSeq*[T](l: List[T]): seq[T] =
  result = newSeq[T](l.size())
  var
    it = l.begin()
  for i in 0 ..< l.size():
    result[i] = it[]
    it = it.next()

• Line 1 - We needed a binding to the std::next method from iterator so that we can increment the iterator starting from the begin() returned value.
• Line 4 - We already know the size of the list object, so we pre-allocated the Nim sequence to the same size by using the newSeq proc.
• Then rest of the code follows the idea presented in the pseudocode above.

Final Code #

The final code to (i) wrap the <list> types and methods and (ii) one method from <iterator>, and (iii) define a proc for converting std::list objects to Nim sequences was only 22 lines! (including comments and blank lines)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

when not defined(cpp): {.error: "This file needs to be compiled with cpp backend: Run 'nim -b:cpp r <file>'.".}

## <list> bindings -- https://en.cppreference.com/w/cpp/container/list
type
  List*[T]                              {.importcpp: "std::list", header: "<list>".} = object
  ListIter*[T]                          {.importcpp: "std::list<'0>::iterator", header: "<list>".} = object

proc initList*[T](): List[T]            {.importcpp: "std::list<'*0>()", constructor, header: "<list>".}
proc size*(l: List): csize_t            {.importcpp: "size", header: "<list>".}
proc begin*[T](l: List[T]): ListIter[T] {.importcpp: "begin", header: "<list>".}
proc `[]`*[T](it: ListIter[T]): T       {.importcpp: "*#", header: "<list>".}

## std::list processing procs and iterators
proc next*[T](it: ListIter[T]; n = 1): ListIter[T] {.importcpp: "next(@)", header: "<iterator>".}

proc toSeq*[T](l: List[T]): seq[T] =
  result = newSeq[T](l.size())
  var
    it = l.begin()
  for i in 0 ..< l.size():
    result[i] = it[]
    it = it.next()

## Test
when isMainModule:
  {.emit: """
  void generateList(int num, std::list<int> *retList)
  {
      for (int i = 0; i < num; ++i) {
          retList->push_back(i * 2);
      }
  }
  """.}
  proc generateList(num: cint; lPtr: ptr List[cint]) {.importcpp: "generateList(@)".}

  var
    l = initList[cint]()
  generateList(10, addr l)
  var
    s = l.toSeq()
  echo "sequence   = ", s
  echo "first elem = ", s[0]
  echo "elem 5     = ", s[5]
  echo "last elem  = ", s[^1]

sequence   = @[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
first elem = 0
elem 5     = 10
last elem  = 18

You can try this code by saving to a list_to_seq.nim file and running nim -b:cpp r list_to_seq.nim.

Alternatively, you can paste this code on https://play.nim-lang.org/, set the ‘Compilation target’ to C++ and hit the yellow ‘Run!’ button.

Summary #

It becomes fairly easy to map any C++ (or C) library to Nim. Once that it done, you can leverage the power of Nim to extend that mapped library or manipulate data generated by that library. And you do not need to be a C++ programmer to do so (I am not).

The folks on Nim Forum, Discord and elsewhere have been very generous with their help and support, and that’s one of the main reasons it’s a joy to use Nim. 🙏

References #

• 📖 Nim Manual – importcpp pragma
• C++ std::list reference
• Nim cppstl  This package contains bindings to std::string, std::vector and std::complex as of <2022-03-10 Thu>. package – can be installed using nimble install cppstl