Clash of the Lambdas

Microbenchmarking Stream APIs of the VM-based, multiparadigm languages: Java 8, Scala, C#, F# on Windows and on Linux.

The results are discussed in our ICOOOLPS'14 workshop paper:

To run the benchmarking suite as is, you will need a system with approximately 3GB of free space for heap allocation. Regarding execution time, a run on a single platform takes approximately 30-40 minutes on an Intel Core i5.

Java 8 (jvm / jmh) sequential parallel

Scala (jvm / jmh) sequential parallel optimized sequential/parallel with ScalaBlitz

C# (clr & mono / Lambda Microbenchmarking 1 ) sequential parallel optimized sequential/parallel with LinqOptimizer

) F# (clr & mono / Lambda Microbenchmarking 1 ) sequential parallel optimized sequential/parallel with LinqOptimizer

)

Setting up on Windows

Install JDK8. Create the JAVA_HOME env variable (for maven to work) and add to Path the JAVA_HOME\bin directory

env variable (for maven to work) and add to the directory Install Gnu Make for Windows and put the bin directory to PATH

directory to Extract Maven and put the bin directory to PATH

directory to Install the C#, F# compilers (if you have Visual Studio 2013 you already have them)

Update the paths in Makefile

Setting up on Ubuntu

Install/Extract JDK8 and update PATH

sudo apt-get install make

sudo apt-get install maven

Follow the instructions on fsharp.org

Running the microbenchmarks

(via a console on Linux o via a Powershell console on Windows)

Compile all tests with make

Clean with make clean

Compile a specific suite with make {java, csharp, fsharp, scala}

Run microbenchmarks with make bench-{java, csharp, fsharp, scala} (e.g., make bench-java ).

Processing the results

If you want to process the unified results we offer a gawk script for linux (that also makes use of the dos2unix command to convert windows result file encoding).

Run the benchmark suite with on both windows and linux with make bench > results.{windows, linux} (any filename), or use it with tee to view the results as it runs with make bench | tee results.{windows, linux}

(any filename), or use it with tee to view the results as it runs with See the unified results for all languages, platforms and benchmarks (the comma separated values are benchmark, mean, mean error, sdev) by running ./process results.linux results.windows > results.processed (make the script executable first).

Team

@biboudis, @NickPalladinos and Y. Smaragdakis

Footnotes: