Eccentric DevelopmentsEleven implementations on the same number of languages have been written so far and the results have been enlightening, to say the least. I'll try to summarize and explain the findings and compare them as fairly as possible. For more info on a particular language, refer to its corresponding blog entry.
The languages explored so far:
The main idea from the first post in the series still remains, I'm comparing programming languages in order to find one that is best suited for rapid prototyping and everyday programming, and also is fun to work with.
For this, I resorted to building a simple path tracer. It might not be the best kind of project to expose the most prominent characteristics of a particular language, but it's the kind of project I like working on in my spare time; that's why I look for expressiveness and speed, with fun factor as a subjective point to consider.
Most of the languages where easy to pickup, at least as much as necessary for this test; except for Haskell, which took me a while to get my head around its pure functional approach, but definitely helped later with quickly understanding OCaml and Elixir.
Compilers and/or interpreters where also easy to install on Ubuntu linux, some needed for new repositories to be added to the package manager (like node.js for example); while Nim, required for the sources to be downloaded, compiled and installed manually.
Before going to the hard data, I must state that working with most of the languages was pretty awesome, building the Haskell implementation was hard but fun; and each has a set of characteristics that makes them unique on their own.
Code Size
All implementations span only a single file, and some metrics from them where put in graphs. The first one shows the number of lines each implementation took to achieve the desired functionality, of rendering a 3D scene.
From the previous graph, we can observe that OCaml needs around a third of the total lines of code needed by C, followed closely by Elixir and Lisp. But this metric doesn't clearly paint the whole picture, so let's take a look at the following graph.
This graph shows the size of the code in bytes, each a character (UTF-8) that you have to type. Here as you can tell, OCaml's advantage is not as pronounced as when we where looking only at lines of code. C takes somewhere around 7500 bytes to express the program, while OCaml takes less than 5000; a difference of 2500 bytes (or characters) doesn't seem as impressive looking exclusively at lines of code, but this only shows that the average length of a line of code in OCaml is bigger than in C.
Nonetheless, OCaml presents itself as the king of expressiveness, followed closely by Nim. Although I was expecting to find a language that needed way less code than C, say, half the size would have been a great finding, it was not the case.
Running Time
This comparison has lots to do with how compilers and interpreters handle the path tracing algorithm implemented. What optimisations they make, and whether to take advantage of the CPU characteristics, like SSE or AVX.
Implementing the same algorithm amongst all the languages was a bit difficult when working with functional programming, but keeping close to the same approach was important to be able to compare the compilers.
Some languages offered the generation of native code, others where simply interpreted and there where a couple that used JIT compilers.
Comparing them one to one, might not be entirely fair, but it helps on determining how much friction you would get while testing and debugging code quickly.
First I'll show you the complete running time comparison graph.
As you can see, the Lua language, using it's regular interpreter offers an abysmal performance; Python, Lua running on LuaJIT, and Elixir aren't doing that well either, taking over 100 minutes to render the image, over 50 times C.
In order to help compare the rest of the implementations, I took away from the graph the previously mentioned languages, and got the following:
How the other languages compare to each other is more easily visible here. Haskell beign the slowest, followed closely by Lisp. Pypy really helped improve the speed of Python making it somewhere around 20 times faster.
C, as the baseline for the comparison, seats almost on top of the graph, surprisingly bested by Nim, although by a very small margin. OCaml sits in third place, running around half the speed of C, not bad at all.
Other languages that I was expecting to perform better, like Go and Haskell, were found to perform worse on terms of efficiency as initially thought. They both come up as disappointments in my book.
On the other hand, Javascript showed up as a big surprise; running almost as fast as Go and faster (by a good margin) than C#. I wonder if this translates linearly to web programming, which would put to shame all those blog posts about Go being a faster alternative to Javascript, running on top of node.js.
Side Note
On later testing, I learned that OCaml's float data type is not single precision, but double (64 bits). So, for a more fair comparison, I took the C and Nim implementations and changed all it's floating point numbers and variables to double precision. The results are shown in the following updated graph.
As can be seen, the change to double precision does have an impact in running time, Nim (Dbl) and OCaml are almost tied on speed. C doesn't have much of an impact on performance, but it gets a little bit slower.
Standings
I have compiled a couple of scatter plots showing how all the languages compare on running time and file size, X and Y axis respectively.
What we are trying to look for, are languages found on the lower leftmost part of the chart.
Because of Lua, Python and Elixir make the majority of the other points look squashed to the left axis, they where removed and the new graph is shown below.
Here we can more easily confirm what was found before, Nim is the fastest language, and OCaml the most expressive one. While Haskell is way on the top right corner; the worst position to be.
Conclusions
Now, you might be inclined to say that Nim is the best language overall, but I would have to disagree on certain points. For starter, Nim is a very recently developed language, almost like a prototype in its current state (until it reaches v1.0), while OCaml has been around for several more years, and it's being actively used in the industry.
But I have to concede that, Nim feels more familiar to a programmer used to procedural and object oriented languages like C and Python. Nim could be categorised as an imperative language with functional touches.
Meanwhile, OCaml feels very different from working with more traditional languages, but doesn't get in your way when you need to use imperative or procedural characteristics; unlike Haskell, on which it is a mayor pain working with random numbers.
Both OCaml and Nim meet the initially stated criteria of: expressiveness, decent speed and fast compilation. Now deciding between those too will get a bit difficult, although I think learning OCaml would be a bigger of a challenge, and as such, a greater learning experience. As a follow up I'm planning to build some more complete projects with Nim and OCaml, to see how their tooling, libraries and all around experience compare, and with that take a more definitive decision.
I'll try to build some more implementations of the path tracer on different languages on a later time, and compare them with the others up to now. Java, Rust, Julia, Perl, Assembler, and many other languages are still around that deserve to be tried.
Follow
You can follow the development of this project on GitHub: https://github.com/niofis/raybench
P.S. Maybe I should just keep with C after all: Scalable C