8

I got myself very surprised after I wrote this small code to compare .NET 4.5 and Java 8 performance in my computer:

class ArrayTest
{
    public int[][] jagged;

    public ArrayTest(int width, int height)
    {
        Height = height;
        Width = width;
        Random rng = new Random();
        jagged = new int[width][];
        for (int i = 0; i < height; i++)
        {
            jagged[i] = new int[width];
            for (int j = 0; j < jagged[i][j]; j++)
            {
                jagged[i][j] = rng.Next(2048);
            }
        }
    }
    public int this[int i, int j]
    {
        get
        {
            return jagged[i][j];
        }
        set
        {
            jagged[i][j] = value;
        }
    }

    public void DoMath(ArrayTest a)
    {
        for (int i = 0; i < Height; i++)
        {
            for (int j = 0; j < Width; j++)
            {
                this[i, j] *= a[i, j];
            }
        }
    }

    public int Height { get; private set; }

    public int Width { get; private set; }
}



class Program
{
    static void Main(string[] args)
    {
        Random rng = new Random();
        const int loop = 10;
        int width = 10000,
            height = 10000;

        ArrayTest a1 = new ArrayTest(width, height),
            a2 = new ArrayTest(width, height);


        Stopwatch sw = new Stopwatch();
        sw.Start();
        for (int i = 0; i < loop; i++)
        {
            a1.DoMath(a2);
        }
        sw.Stop();

        Console.WriteLine("Time taken: " + sw.ElapsedMilliseconds);

        Console.ReadKey();
    }
}

Here is the Java version:

    public class ArrayTest {
    private int width, height;
    private int[][] array;

    public ArrayTest(int width, int height) {
        this.width = width;
        this.height = height;
        array = new int[height][width];
        Random rng = new Random();
        for (int i = 0; i < height; i++) {
            for (int j = 0; j < width; j++) {
                array[i][j] = rng.nextInt(2048);
            }
        }
    }

    public int getWidth() {
        return width;
    }
    public void setWidth(int width) {
        this.width = width;
    }
    public int getHeight() {
        return height;
    }
    public void setHeight(int height) {
        this.height = height;
    }
    public int[][] getArray() {
        return array;
    }

    public void doMath(ArrayTest a) {
        for (int i = 0; i < height; i++) {
            for (int j = 0; j < width; j++) {
                array[i][j] *= a.getArray()[i][j];
            }
        }
    }

}

public class Main {

    public static void main(String[] args) {
        // TODO Auto-generated method stub
        final int loops = 10;
        int width = 10000, height = 10000;
        ArrayTest a1 = new ArrayTest(width, height),
                a2 = new ArrayTest(width, height);

        long start, end;


        start = java.lang.System.currentTimeMillis();
        for (int i = 0; i < loops; i++) {
            a1.doMath(a2);
        }
        end = java.lang.System.currentTimeMillis();
        System.out.println("Elapsed time: " + (float)(end - start));
    }

}

In my computer this C# code is taking about 5200ms to run, and the Java version is taking about 2800ms(!!!). I was actually expecting that the .NET version would run faster (or at least close to) than Java, but got very surprised with this result.

Note: I ran the .NET version compiled in release mode, outside Visual Studio.

Can somebody explain this result? Is this really right? How could I rewrite this code so that the C#.NET version gets closer to the Java one in execution speed?

[edit]

Well, I know this is not a really valid benchmarking or a fair comparison, but how could I rewrite this code so that the C#.NET version gets closer to the Java one in execution speed? Tried in any forms (manipulating the jagged array directly, via a getter, etc), but the test ran even slower.

[edit 2]

Edited the test so that I have width and height = 500, and loop = 5000. Now I get about 6300ms for .NET version, and about 3700ms for Java version. ran the test multiple times for each version, of course.

[edit 3]

Just wrote a similar test using flat arrays instead of 2D arrays, and this time the .NET version runs equally or even faster than Java. So is that it? C#.NET's jagged arrays are just slower than Java's multidimensional arrays?

13
  • 1
    I'm surprised, too! Q: Could you post the Java version? Commented Apr 29, 2015 at 3:51
  • 5
    You need to do proper benchmarking to get meaningful results, but the test isn't fair anyway. In the java version you're doing array[i][j] *= a.getArray()[i][j]; which only accesses the array once if I understand compound assignment correctly. In the C# version you're using indexers, so you're not accessing the array directly but via the get and set methods. Commented Apr 29, 2015 at 4:15
  • 2
    There are lots of reasons such a difference could exist. For sure, running just 10 loops on each, you're mostly testing the JIT compiler, not the code itself. So the test itself is not even valid. Even if a valid test showed a difference, there is still the question of writing in each language using the appropriate techniques. Apples-to-apples comparisons are very difficult to get right, and this one definitely isn't. Commented Apr 29, 2015 at 4:16
  • 2
    This jagged = new int[width][]; could have caused an IndexOutOfRangeException if you had not chosen width and height to be equal Commented Apr 29, 2015 at 4:22
  • 3
    AFAIK the Java compiler is more "aggressive" with unrolling loops which can make a huge difference for short loops. I have seen an example in which Java actually outperformed native code (written in C++) in a textbook implementation of the sieve of Eratosthenes. So you may want to try more iterations as well. That still won't allow a general judgement over the performance of Java vs. C#/.NET of course. Commented Apr 29, 2015 at 6:29

2 Answers 2

11

Any time you do any kind of benchmarking or performance analysis, you need to ask a lot of questions, and take any particular result with (to paraphrase Tanenbaum), "a metric ton of salt".

HOWEVER ....

I copied pasted your code, and this is what I got:

Compiler      Version   Timing
--------      -------   ------
MSVS 2012     C# 5      4448.0
Eclipse Luna  JRE 1.7    977.0

So the Java program ran 4.5x faster than the C# program.

I also ran the MSVS "Profiling Wizard":

https://msdn.microsoft.com/en-us/library/dd264959.aspx

MSVS Profiler Summary Large picture

As you might be able to see from the screenshot, the "Big Pig" in this profile was ArrayTest.get_item(int32, int32), which consumed NEARLY HALF of the execution time:

enter image description here Large picture

Sign up to request clarification or add additional context in comments.

3 Comments

Whoa! Can you rewrite this test using direct array access and test it in the profiler and tell me what happens? In my case, the test runs even slower, and I don't have VS Pro os any profiler to see what's going on.
We are trying to compare direct access in Java with pointer access in C#. In few words in order to store/load the next value of array java uses plain mov with pointer bump inc, c# uses more expensive operation lea which compute effective address of value on each load/store! I have changed the test to direct access and got the same results for java and C#
Well, the code got a little faster, but it wasn't enough. I had to make several changes in the way I was looping through the 2d array, so that I could get a result equal to that of Java.
1

You can't judge the performance based on single instance. You need to do it for multiple instance. If you want concrete answers on performance I would suggest you to read quora, blog or SO.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.