Java vs. C execution time

[duragezic]

Our senior design project is basically a feasibility study of Real Time Java on Embedded Systems. Real time capabilities aside, one task we have to do is write a program in Java, write the same program in C, and time how long it takes to execute. The program does bilinear interpolation which is just sort of a number crunching thing.

I realize many factors can affect the performance. The program tested in Java and C has no real time constraints and is single threaded. It simply performs bilinear interpolation for a specified number of runs. We time the total time spent on all of the runs and divide it by the number of calculations to get the time per run.

I don't remember specific numbers, but we ended up with C being something like 315 times faster than Java. That is huge! We asked the company who did the RTJ implementation for us how much we can expect C to be faster and were told that with fully compiled (not interpreted) Java code, C will be approximately 3 times faster. We have not yet tested with compiled code since we do not have enough Flash memory to compile everything, so we are in the process of profiling and compiling only a portion of the Java code.

I concerned that even with compiled code, our Java program is going to be many many times slower than C. I've searched and found a few numbers thrown around, which was that interpreted Java code is 10-20x slower than C.

Has anyone done any work with this and have a rough idea? I've looked at the code to determine if there is something happening with the Java VM that would cause it to be so much slower. I think pretty much all of the variables used are on the stack. I think the garbage collector only runs when references to objects on the heap are no longer referenced, something that I don't think we are doing. Any ideas on things that could be happening with the JVM to cause such a slowdown, or things to avoid doing? Thanks

Java code

//see http://en.wikipedia.org/wiki/Bilinear_interpolation
public class NewLinear {
public static void main(String[] args) {
//x axis values
float xBegin = 0;
float xIncrement = 1;
int xLength = 10;

//y axis values
float yBegin = 0;
float yIncrement = 1;
int yLength = 10;

//low & high limits of possible random numbers of value matrix
int qLow = 0;
int qHigh = 10000000;

//number of times to run interpolation (random numbers will
//be used for x & y values to interpolate)
int numTimesRun = 10000;

//build x & y arrays according to user input
float[] xArray = build1DArray(xBegin, xIncrement, xLength);
float[] yArray = build1DArray(yBegin, yIncrement, yLength);

//build q array
float [][] qArray = build2DArray(xLength, yLength, qLow, qHigh);

printComplete(qArray, xArray, yArray);
System.out.println("Timer Started");

for ( int j = 0; j < 30; j++ ) {
long startTimeMilli = System.currentTimeMillis();

//prompt user for x values & linearly interpolate y values
for ( int i = 0; i < numTimesRun; i++ ) {

//create random x & y values between the high & low x & y axis values
//make max values less than max x & y axis values
float xMax = ((xIncrement * (xLength-1)) + (xBegin)) - 0.00001F;
float yMax = ((yIncrement * (yLength-1)) + (yBegin)) - 0.00001F;

float xVal = ((float)Math.random() * xMax) - xBegin;
float yVal = ((float)Math.random() * yMax) - yBegin;

//find y value of x value user inputted
float qVal = interpolate(xArray, yArray, qArray,
xVal, yVal,
xLength, yLength,
xIncrement, yIncrement,
xBegin, yBegin);
}
long endTimeMilli = System.currentTimeMillis();
System.out.println("and " + (endTimeMilli - startTimeMilli) + " milliseconds");
System.out.println("for an average of " + (endTimeMilli - startTimeMilli)/(long)numTimesRun + " milliseconds");
}
}

public static float[] build1DArray(float xBegin, float xIncrement, int xLength) {
float[] array = new float[xLength];
float xVal = xBegin;

for (int x = 0; x < array.length; x++){
array[x] = xVal;
xVal = xVal + xIncrement;
}

return array;
}

public static float[][] build2DArray(int xLength, int yLength, float min, float max) {
float[][] array = new float[yLength][xLength];

for (int row = 0; row < array.length; row++){
for(int col = 0; col < array[0].length; col++) {
float n = (float)(max * Math.random()) + min;

array[row][col] = n;
}
}
return array;
}

public static void print2DArray(float[][] array) {
for (int row = 0; row < array.length; row++){
for(int col = 0; col < array[0].length; col++) {
System.out.print(array[row][col] + "\t ");
}
System.out.println();
}
System.out.println();
}

public static void print1DArray(float[] array) {
System.out.print("[ ");
for(int j=0;j<array.length;j++) {
System.out.print(array[j]);
if ( j != array.length-1 ) {
System.out.print("\t");
}
}
System.out.println(" ]\n");
}

public static void printComplete(float[][] qArray, float[] xArray, float[] yArray) {
System.out.println("y");
for (int row = 0; row < qArray.length; row++){
for(int col = 0; col < qArray[0].length; col++) {
if (col == 0) {
System.out.print(yArray[yArray.length-row-1] + "\t ");
}
System.out.print(qArray[row][col] + "\t ");
}
System.out.println();
}
System.out.print("x\t");
for(int j=0;j<xArray.length;j++) {
System.out.print(xArray[j]);
if ( j != xArray.length-1 ) {
System.out.print("\t");
}
}
System.out.println("\n");
}

/**
* A function that uses bilinear interpolation to find an estimated value
* given an x & y value
*
* @param xArray x-axis numbers
* @param yArray y-axis numbers
* @param qArray matrix built relating to x-axis & y-axis numbers
* @param xVal x value of user input
* @param yVal y value of user input
* @param xLength length of x axis
* @param yLength length of y axis
* @param xIncrement value x axis is incremented by
* @param yIncrement value y axis is incremented by
* @param xBegin value x axis begins at
* @param yBegin value y axis begins at
* @return bi-linearly interpolated value
*/
public static float interpolate(
float[] xArray, float[] yArray, float[][] qArray,
float xVal, float yVal,
int xLength, int yLength,
float xIncrement, float yIncrement,
float xBegin, float yBegin) {

//Make sure x & y pair not in table; if it is return corresponding q value
for (int x = 0; x < xArray.length; x++){
if ( xArray[x] == xVal ) {
//now check for y vals
for ( int y = 0; y < yArray.length; y++ ) {
if ( yArray[y] == yVal ) {
System.out.println("x & y pair in table");
return qArray[(yLength-1)-y][x];
}
}
}
}

//Find the index of the table entry just below & above the x val & y val
//then set variables used in formula
int xIndex = (int)((xVal - xBegin) / xIncrement);
int yIndex = (int)((yVal - yBegin) / yIncrement);
float x1 = xArray[xIndex];
float x2 = xArray[xIndex + 1];
float y1 = yArray[yIndex];
float y2 = yArray[yIndex + 1];

float x1y1 = qArray[(yLength-1)-yIndex][xIndex];
float x1y2 = qArray[(yLength-1)-(yIndex+1)][xIndex];
float x2y1 = qArray[(yLength-1)-yIndex][xIndex+1];
float x2y2 = qArray[(yLength-1)-(yIndex+1)][xIndex+1];

//perform bilinear interpolation
float r1 = ( ((x2 - xVal)/xIncrement) * x1y1 ) + (((xVal-x1)/xIncrement) * x2y1);
float r2 = ( ((x2 - xVal)/xIncrement) * x1y2 ) + (((xVal-x1)/xIncrement) * x2y2);
float interp = ( ((y2 - yVal)/yIncrement) * r1 ) + (((yVal-y1)/yIncrement) * r2);

return interp;
}
}

C code

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

void build1DArray(float, float, int, float *);
void build2DArray(int, int, float, float, float **);
void printComplete(float**, float*, float*, int, int);
float interpolate(
float* , float*, float**,
float, float,
int, int,
float, float,
float, float);

int main(void) {
float xBegin = 0;
float xIncrement = 1;
int xLength = 10;
int i = 0;
int j = 0;

//y axis values
float yBegin = 0;
float yIncrement = 1;
int yLength = 10;

//low & high limits of possible random numbers of value matrix
int qLow = 0;
int qHigh = 10000000;

//number of times to run interpolation (random numbers will
//be used for x & y values to interpolate)
int numTimesRun = 1000000;
float *xArray;
float *yArray;
float **qArray;

int startTime;
int endTime;

//build x & y arrays according to user input
xArray = malloc(xLength * sizeof (float)); //new float[xLength];
build1DArray(xBegin, xIncrement, xLength, xArray);
yArray = malloc(yLength * sizeof (float)); //new float[yLength];
build1DArray(yBegin, yIncrement, yLength, yArray);

//build q array
qArray = (float **) malloc(yLength*sizeof(float)); //new float* [yLength];
for(i=0;i<yLength;i++)
*(qArray+i) = malloc(xLength*sizeof(float)); //new float[xLength];
build2DArray(xLength, yLength, qLow, qHigh, qArray);

printComplete(qArray, xArray, yArray, xLength, yLength);

for (j = 0; j < 30; j++ ) {
startTime = clock();

//prompt user for x values & linearly interpolate y values
for (i = 0; i < numTimesRun; i++ ) {

//create random x & y values between the high & low x & y axis values
//make max values less than max x & y axis values
float xMax = ((xIncrement * (xLength-1)) + (xBegin)) - 0.00001;
float yMax = ((yIncrement * (yLength-1)) + (yBegin)) - 0.00001;

float xVal = (rand()/((float)(RAND_MAX)+1) * xMax) - xBegin;
float yVal = (rand()/((float)(RAND_MAX)+1) * yMax) - yBegin;

//find y value of x value user inputted
float qVal = interpolate(xArray, yArray, qArray,
xVal, yVal,
xLength, yLength,
xIncrement, yIncrement,
xBegin, yBegin);
}
endTime = clock();

printf("Loop %d: ",j);
printf("Done, took: %f seconds\n", (float)((endTime - startTime)/CLOCKS_PER_SEC));
printf("Done, took: %f milliseconds\n", (float)((endTime - startTime)/CLOCKS_PER_SEC)/(float)numTimesRun*1000.0);
}
free(xArray);
free(yArray);
for(j=0;j<yLength;j++)
free(qArray[j]);
free(qArray);
return 0;
}

void build1DArray(float xBegin, float xIncrement, int xLength, float *array) {
float xVal = xBegin;
int x = 0;

for (x = 0; x < xLength; x++){
array[x] = xVal;
xVal = xVal + xIncrement;
}

return;
}

void build2DArray(int xLength, int yLength, float min, float max, float **array) {
int row = 0;
int col = 0;

for (row = 0; row < yLength; row++){
for(col = 0; col < xLength; col++) {
//fill with random number between yLow and yHigh
float n = (float)(max * rand()/((float)(RAND_MAX)+1)) + min;

array[row][col] = n;
}
}
return;
}

void printComplete(float** qArray, float* xArray, float* yArray, int xLength, int yLength) {
int row = 0;
int col = 0;
int i = 0;
printf("y\n");
for (row = 0; row < yLength; row++){
for(col = 0; col < xLength; col++) {
if (col == 0) {
printf("%.0f\t",yArray[yLength-row-1]);
}
printf("%f \t",qArray[row][col]);
}
printf("\n");
}
printf("x\t");
for(j=0;j<xLength;j++) {
printf("%.0f",xArray[j]);
if ( j != xLength-1 ) {
printf("\t\t");
}
}
printf("\n");
}

float interpolate(
float* xArray, float* yArray, float** qArray,
float xVal, float yVal,
int xLength, int yLength,
float xIncrement, float yIncrement,
float xBegin, float yBegin) {

int x,y;
int xIndex;
int yIndex;
float x1;
float x2;
float y1;
float y2;

float x1y1;
float x1y2;
float x2y1;
float x2y2;
float r1,r2,interp;

//Make sure x & y pair not in table; if it is return corresponding q value
for (x = 0; x < xLength; x++){
if ( xArray[x] == xVal ) {
//now check for y vals
for (y = 0; y < yLength; y++ ) {
if ( yArray[y] == yVal ) {
return qArray[(yLength-1)-y][x];
}
}
}
}

//Find the index of the table entry just below & above the x val & y val
//then set variables used in formula
xIndex = (int)((xVal - xBegin) / xIncrement);
yIndex = (int)((yVal - yBegin) / yIncrement);
x1 = xArray[xIndex];
x2 = xArray[xIndex + 1];
y1 = yArray[yIndex];
y2 = yArray[yIndex + 1];

x1y1 = qArray[(yLength-1)-yIndex][xIndex];
x1y2 = qArray[(yLength-1)-(yIndex+1)][xIndex];
x2y1 = qArray[(yLength-1)-yIndex][xIndex+1];
x2y2 = qArray[(yLength-1)-(yIndex+1)][xIndex+1];

//perform bilinear interpolation
r1 = ( ((x2 - xVal)/xIncrement) * x1y1 ) + (((xVal-x1)/xIncrement) * x2y1);
r2 = ( ((x2 - xVal)/xIncrement) * x1y2 ) + (((xVal-x1)/xIncrement) * x2y2);
interp = ( ((y2 - yVal)/yIncrement) * r1 ) + (((yVal-y1)/yIncrement) * r2);

return interp;
}

 

[postmortemIA]

are you allocating enough memory to java program? you should use some java profiler to see if garbage collection is being called.

also I wonder how you manage to keep everything on stack, don't you have something that is not primitive variable.

 

[IHateMyJob2004]

write a program in Java, write the same program in C.

Thee is your first problem.... make the source open to the public for review.

In the real time world, you can do things like instantiating objects and when they are no longer needed, simply reuse them. Do thi in your Java code and not your C code and Java will kill C. And visa-versa.

Who wrote the real time VM?

And based on how java does things internally, I wouldn't use it for a real tiem embedded system. The military simply doesn't allow it.

Gotta run.

 

[duragezic]

The total memory we have allocated is pretty much the maximum available. Initially, we did not allocate enough and got a runtime error, but we increased it till it worked. I could play around with for example lowering the heap and increasing the stack, but the total is pretty much maxed.

I'm unsure on the exact capability of the profiler we have. We asked a company rep about knowing when the garbage collector is called, and he said there wasn't really any way to know. Need to work with the profiler more to see if it can help.

Actually, there is three arrays on the heap I think, since they are created with "new", but they are very very small. Everything else is local variables (integers and floats). We tested it as a single method instead of repeated calls to other methods, but the time difference was so small I don't think it matters.

So yes, I think it is all primitive types although I guess arrays are Objects. The timer only times the calculations, not any initialization stuff. Timing things in C is not as simple as in Java, and even if the times aren't exact, you can just tell in wall clock time, like say the C one can run 10,000,000 iterations in the 10 seconds required for Java to run 100,000 iterations.

edit: I will post the source code tomorow.

This particular test doesn't use any Java strengths, objects, etc... it is just pretty much testing raw performance. I wouldn't think array accesses, random number generation, etc could be that much different in Java than in C, but something is up here that is killing Java.

The software package is Scorpion from DDC-I. It seems to use the JamaicaVM from aicas Gmbh.

Yeah, hear ya about Java and RT embedded systems. Heard it many times actually. So the outcome of this isn't expected to be "omg Java pwns!", this just happens to be something of interest to a corporation, so they sponsored it as a senior design project at my school.

 

[brikis98]

comparing the "speeds" of two programming languages is anything but easy. Depending on the problem, the implementation and who you ask, you'll hear different opinions. Some say running interpreted code via a VM must be slower. Some just like to say that Java is slow, which used to be true, but is no longer the case. In fact, some would argue that the massive amount of optimizations that Java can do make it way faster. Most of the talk I've heard recently strongly favors Java, and each new release seems to be noticeably faster than the last. However, I'm not sure how that applies to real time environments.

at any rate, here are some sites with more info:

link 1
link 2
link 3
link 4
link 5
link 6

 

[postmortemIA]

duragezic, you can use regular java profiler and rand program on desktop JVM, after all it is java code...

there's JProfiler, and NetBeans IDE comes with cool one that is free.

 

[tfinch2]

There is a tradeoff between the level of abstraction and performance. More likely than not you can't have your cake and eat it too.

 

[IHateMyJob2004]

A good VM will in many cases run faster than C/C++ code. I'll just leave it at that.

 

[degibson]

Question: If you've implemented both programs, C and Java, and found that C is way faster, why aren't you just using the C code?

1) Everyone who pointed out that testing the performance differences between two languages in this way is hard gets a /nod in my book. Implementing the same algorithm in the same way in different languages doesn't optimize either case. This leads to my next point...
2) C faster than Java is truth. A well-tuned C program will always outperform a well-tuned Java program. Emphasis on the well-tuned. Commonly, an untuned C program is faster than a tuned Java program, but thats beside the point.
3) Java is great for its other features -- programmability, portability, flexibility, you name it. C is a sledgehammer -- macros around raw ASM. If you're building beautiful GUIs and such, use Java. If you're tearing down a wall, use C.
4) In the context of realtime requirements, performance concerns aren't faster is better, they are fast enough is fast enough. So, is Java fast enough?

Edit: Let the pro-Java folks ensue their anti #2 rant. =)

 

[brikis98]

Originally posted by: degibson
Question: If you've implemented both programs, C and Java, and found that C is way faster, why aren't you just using the C code?

1) Everyone who pointed out that testing the performance differences between two languages in this way is hard gets a /nod in my book. Implementing the same algorithm in the same way in different languages doesn't optimize either case. This leads to my next point...
2) C faster than Java is truth. A well-tuned C program will always outperform a well-tuned Java program. Emphasis on the well-tuned. Commonly, an untuned C program is faster than a tuned Java program, but thats beside the point.
3) Java is great for its other features -- programmability, portability, flexibility, you name it. C is a sledgehammer -- macros around raw ASM. If you're building beautiful GUIs and such, use Java. If you're tearing down a wall, use C.
4) In the context of realtime requirements, performance concerns aren't faster is better, they are fast enough is fast enough. So, is Java fast enough?

Edit: Let the pro-Java folks ensue their anti #2 rant. =)

no rant. just show some proof for #2. i posted a large number of links in my reply, and a surprising number of them show Java at least as good as C, if not better. so... what are you basing #2 on? a gut feeling? the obsolete impression that java is just generally slow?

 

[Markbnj]

Originally posted by: brikis98

Originally posted by: degibson
Question: If you've implemented both programs, C and Java, and found that C is way faster, why aren't you just using the C code?

1) Everyone who pointed out that testing the performance differences between two languages in this way is hard gets a /nod in my book. Implementing the same algorithm in the same way in different languages doesn't optimize either case. This leads to my next point...
2) C faster than Java is truth. A well-tuned C program will always outperform a well-tuned Java program. Emphasis on the well-tuned. Commonly, an untuned C program is faster than a tuned Java program, but thats beside the point.
3) Java is great for its other features -- programmability, portability, flexibility, you name it. C is a sledgehammer -- macros around raw ASM. If you're building beautiful GUIs and such, use Java. If you're tearing down a wall, use C.
4) In the context of realtime requirements, performance concerns aren't faster is better, they are fast enough is fast enough. So, is Java fast enough?

Edit: Let the pro-Java folks ensue their anti #2 rant. =)

no rant. just show some proof for #2. i posted a large number of links in my reply, and a surprising number of them show Java at least as good as C, if not better. so... what are you basing #2 on? a gut feeling? the obsolete impression that java is just generally slow?

I haven't had time to look through the links, but the key question will always be whether the tester has realistically simulated identical loads on each program. C code compiles down to machine language object modules that execute with a simple jmp to an offset. No background tasks, no garbage collector, no thread management, etc. So how do you write a C program that accurately mimics Java GC, or object reference management?

We used to have the same argument about C vs. C++. Code a C program to have all the encapsulation and modularity of a C++ program and they'll run about the same speed, was the way I most often heard people put it. But I don't think it's a very useful comparison. Managed languages are simply safer, and easier to teach people to use, and they're the future of computing, without a doubt. C is likely faster in most scenarios, but outside of the OP's world of embedded systems few people still care, and even in that world, over time, the abundance of machine resources will make the argument obsolete.

 

[DaveSimmons]

Originally posted by: brikis98
no rant. just show some proof for #2. i posted a large number of links in my reply, and a surprising number of them show Java at least as good as C, if not better. so... what are you basing #2 on? a gut feeling? the obsolete impression that java is just generally slow?

I've just looked at link #1, but it only shows optimized java could someday be faster than untuned C/C++, such as this:
for( int i = 0; i < len ; i++ ) { a[ I ] = ... }

Where len is for some reason (such as being a pointer in the C/C++ code) identifiable as constant to the JVM but not to a C/C++ compiler.

The problem with that argument is that if this was time-sensitiive code no decent C/C++ coder would have len be a pointer instead of a local variable on the stack.

The same with the "garbarge collection could be faster than malloc" argument in that link. For time-sensitive code a decent programmer will avoid multiple mallocs by allocating one large buffer and/or by re-using memory.


In most programs there are large sections where optimization doesn't help because the code is waiting on the user or the operating system (file I/O, display drivers) so if a JVM beats sloppy C/C++ code there it doesn't really matter.


It's true though that there will be some parts of some program where performance matters and where you might get faster code from java if no effort is put into tuning the code in C/C++. That is a win for java, to get decent speed with less effort. It's the same argument for why a good C/C++ compiler can be better than writing everything in assembly by hand.

 

[duragezic]

Well I updated the OP with the code for each program. It is ugly. Its a bummer the Attach Code feature got broken and has not been fixed yet. And using Quote is pretty awful due to losing spaces. So I realize that no one wants to read that. I didn't even write the code, I just posted it to show the type of operations that are being done. Whatever is being done in Java is pretty much converted line-by-line to C as needed. So if there is something like a huge inefficiency in the Java code, it should be in the C code as well, or maybe that could be true but it is affecting Java performance far more?


Without going into a lengthy reply right now, thank you for the input.

I guess my answer in regards to some of the questions is this:

Whether or not I could tune up the Java code and use sloppy C code to try and make Java faster, that is not really the point. I know that Java will be slower, it'd be weird to find it faster, considering the company told us to expect C to be several times faster than compiled Java code. It is just that one of the requirements is to do a raw performance comparison like this.

Without having results for compiled code yet, I guess I just don't believe that compiling instead of interpreting above code could would change the 314x faster to something expected like 3-5x faster. Maybe I shouldn't be assuming that, it just makes me wonder if there is some extremely inefficient Java code or something.

 

[kylef]

Under most circumstances, Java programs have high startup latency. This latency tends to dominate runtime for short-lived programs like command-line utilities. For longer-run programs (i.e., services which run for days on a server) this latency is insignificant. But Java also imposes significant type system overhead as well, probably at least as much as C++ over C. JVMs try to make up for this overhead by performing run-time optimizations which compilers can't do. Well-written JVMs are very successful and have honestly shortened the performance gap considerably.

With that said, most of the "Java performance" comparisons I have seen always seem to choose really stupid benchmark programs which happen to highlight JVM optimizations. In number-crunching applications, run-time VM subexpression elimination and other optimizations can have significant perf benefits, giving Java an edge. For object creation benchmarks, coders usually compare the creation of relatively small, uniform objects, and here the JVM does a much better job at heap management than the default heap allocators provided by CRTs. Other examples are similarly trivial, like a tight nested loop (which the JVM can analyze at runtime and unroll in nearly all circumstances), or tons of function calls, etc. Good JVMs can usually optimize all of these artificial scenarios quite well. What's not so clear is that these scenarios occur very frequently in real-world programs.

The big key takeaway, however, is that Java performance is heavily dependent on the JVM implementation. Even when using Sun's JVM, if you switch from the client to the server JVM you'll see completely different perf results because of the different tunings and optimizations. JVM implementations tend to be very inconsistent from platform to platform, and even version to version.

The other point here is that "optimized code" will usually be faster in C than in Java. Why? Java doesn't offer low-level optimizations; it explicitly avoids them. In C, you can do things like write your own heap management algorithm to store your app's data. Or optimize I/O for the type of underlying device on which you're operating. Or exploit some new instruction set on the CPU you're targeting. In Java you have to trust the JVM and the class libraries to do this for you, because the whole point is to virtualize the machine so that the programmer doesn't care. This discrepancy almost NEVER shows up in the "performance comparisons", because they always focus on comparing performance for apps written with virtually equivalent code. If you turn a C programmer loose and tell them, "optimize away", you're probably going to see different results.

Of course, all such generalizations are imprecise. You really only "know" if Program A is faster than Program B if you time them explicitly. And that means you have to write "comparable" programs first. And for substantial programs, few people are willing to implement them in C, C++, and Java just to check perf! In fact, it's not clear to me that this has ever really happened.

 

[duragezic]

Hmm, interesting. Well one thing I should point out, and I'm at my home computer so I don't know the exact words, but this particular implementation requires us to check a box something like "Run-time checks disabled", which among other boxes we were told we had to do in order for it to work (or fit, I don't remember) on the board. I wondered about that a little while ago but forgot to look into it more, cause I don't know that seems kind of crappy, to be unable to have those of the runtime checks, a desirable trait of Java, and a sort of argument as to why a VM can be a good thing?

I guess I need to get at least some of that Java code compiled and see the results. I don't expect some of these skewed results from companies pushing Java that their JVM is actually faster than C by some huge percentage (just like benchmarks for other computer hardware and software, with the right conditions you can show what you want in a positive light), I still just don't feel confident in reporting that with this JVM running this program, C is 300x faster. Again, just sort of feels like something else is at work here that I don't know about.

In regards to your last two paragraphs, I see what you mean. The only reason we did we what we did is that was what was given to us. If we figured out something better, I'm sure we could suggest changes and that would probably be okay if it made the comparison more fair, and by more fair I mean more accurately approximating real-world performance. Maybe this whole binlinear interpolation will never lead to a good comparison, or maybe it just needs optimizations to each side so that they aren't basically identical.

Heh, still though, even with everything said taken into consideration, the result still seems a bit funky.

 

[degibson]

Originally posted by: brikis98

Originally posted by: degibson
...
2) C faster than Java is truth. A well-tuned C program will always outperform a well-tuned Java program. Emphasis on the well-tuned. Commonly, an untuned C program is faster than a tuned Java program, but thats beside the point.
...

Edit: Let the pro-Java folks ensue their anti #2 rant. =)

no rant. just show some proof for #2. i posted a large number of links in my reply, and a surprising number of them show Java at least as good as C, if not better. so... what are you basing #2 on? a gut feeling? the obsolete impression that java is just generally slow?

I don't have the inclination to find links, admittedly (kudos by the way), mostly because I am unfamiliar with others' methodology and don't trust performance evaluations anyway. I am basing my claim on an inductive hypothesis.

Consider a task, which can be optimally represented as a sequence of low-level machine instructions (this is a big leap -- assume an optimal solution exists on a given HW platform. This sequence varies by HW).

A well-tuned Java implementation of this task (e.g., one that comes as close as possible within the bounds of the Java language to the optimal sequence of dynamic instructions) will always contain more instructions, as observed by the hardware, than an equally well-tuned C version, if only due to the overhead of the JVM in the first place. Even with a prophetic JVM that JITs the entire code, there is still the JIT overhead to consider.

As an aside, I also assert that as long as you can play tricks with the JVM, it is fair game to play tricks with the C-language runtime as well -- exotic linking and such.

 

[degibson]

Back to comments about the code:

I see you're calling Cstdlib's rand() and Java's Math.random() A LOT. Try profiling the execution time of just these functions and see how they compare on your system. I have found the performance of rand() on various platforms to be disappointing, as well as the degrees of randomness (I don't know about Math.random()). If there is a large performance difference, implement your own random number generator and use it.

 

[kylef]

Originally posted by: degibson
Consider a task, which can be optimally represented as a sequence of low-level machine instructions (this is a big leap -- assume an optimal solution exists on a given HW platform. This sequence varies by HW).

A well-tuned Java implementation of this task (e.g., one that comes as close as possible within the bounds of the Java language to the optimal sequence of dynamic instructions) will always contain more instructions, as observed by the hardware, than an equally well-tuned C version, if only due to the overhead of the JVM in the first place. Even with a prophetic JVM that JITs the entire code, there is still the JIT overhead to consider.

Not so sure about the validity of your argument here. There are certainly well-known circumstances where interpreters and/or VMs can perform substantial run-time optimizations that are off-limits to compilers, because compilers are not able to make assumptions about run-time data. (Example: Eliminating range checks, null checks, common sub-expressions by merging parse trees, etc.) You can fairly easily design programs which explicitly capitalize on these run-time optimizations, and they will *clearly* outperform similar implementations in C. Not all will, but some certainly will.

Now of course it is possible to re-write the C program to achieve similar performance, but you would basically be re-implementing the JVM optimizations inside your program, at a dramatic cost in terms of maintainability, readability, complexity, and size. (For example, you can certainly eliminate range checking and NULL checking in your code and see performance benefits, but what do you lose in exchange? Usually reliability and maintainability.)

So in some circumstances, all these fancy JVM optimizations can offset the extra overhead of using a JVM in the first place.

But what Java will never allow you to do is optimize for hardware. Code which is optimized for hardware will almost always beat code that isn't, JVM included.

 

[Markbnj]

Now of course it is possible to re-write the C program to achieve similar performance, but you would basically be re-implementing the JVM optimizations inside your program, at a dramatic cost in terms of maintainability, readability, complexity, and size. (For example, you can certainly eliminate range checking and NULL checking in your code and see performance benefits, but what do you lose in exchange? Usually reliability and maintainability.)

You have to, as much as possible, write each version of the program to be cross-tested so that it is as modular, maintainable, etc., as the other. Theoretically the C++ code could be much faster just because the memory management is hand-coded, but the Java guy could say "Yeah, but look how much more fragile it is over time!" The whole debate devolves into meaningless semantic wrangling.

On the other end of the spectrum you can test things like searching a binary tree, or building a huge XML document from the root down, and then deallocating it. But in the end those tests aren't that meaningful either.

I think it's a fascinating conversation, and I have my popcorn to hand, but it just doesn't matter much. The future is virtualization, managed code, managed memory, managed everything, because it has to be.

 

[degibson]

Originally posted by: kylef

Now of course it is possible to re-write the C program to achieve similar performance, but you would basically be re-implementing the JVM optimizations inside your program, at a dramatic cost in terms of maintainability, readability, complexity, and size. (For example, you can certainly eliminate range checking and NULL checking in your code and see performance benefits, but what do you lose in exchange? Usually reliability and maintainability.)

So in some circumstances, all these fancy JVM optimizations can offset the extra overhead of using a JVM in the first place.

I was assuming, as you suggested, that runtime checks were removed in favor of performance. Thats why I clearly said well-tuned. Performance tuning often involves removing superfluous runtime checks and defensive code, often to the detriment of reliability and maintainability. Sometimes there are better / more general solutions to this problem, sometimes there aren't. However, this thread isn't about reliability or maintainability -- its about performance and performance sometimes requires more effort than just compiler optimization.

But what Java will never allow you to do is optimize for hardware. Code which is optimized for hardware will almost always beat code that isn't, JVM included.

There are varying degrees of this, and I think you're selling Java a little short, in fact! JVMs that are themselves compiled with platform optimizations could be said to be 'optimized for the hardware'. If, on the other hand, you're talking about using inline ASM or some other voodoo, you're quite right. It is arguable how 'hardware optimization' you should use in some cases, but this thread is about performance after all.

Originally posted by: Markbnj
The future is virtualization, managed code, managed memory, managed everything, because it has to be.

The future for most application programmers probably is virtualization, managed shoelaces, etc., and I'm thankful for that. However, there will still be 'guru-level' applications that look a lot more like C, and of course, something blazing fast has to be doing all the managing, virtualization, etc. in the first place.

 

[Markbnj]

The future for most application programmers probably is virtualization, managed shoelaces, etc., and I'm thankful for that. However, there will still be 'guru-level' applications that look a lot more like C, and of course, something blazing fast has to be doing all the managing, virtualization, etc. in the first place.

Something like the CLR/VM will eventually be running in firmware. You're right that somewhere, somehow, some small group of people will always be writing closer to the machine, but it will be a reall small group compared to the masses of application programmers.

 

[kylef]

I think the next big step towards a managed world will be the replacement of the Windows subsystem (Win32) by a new version of Microsoft's CLR implemented directly on top of native NT APIs. Basically, eliminating a layer of the OS underneath .NET and ditching Win32 once and for all. Obviously it would stick around as a "compatibility" subsystem, kind of like Mac OS Classic was on OS X originally.

 

[duragezic]

Well just to update... ended up compiling almost all of the Java code, tried with and without the Math.random() statement, and there was a few other things. Got Java to run 7.8 times slower than C and estimated that additional optimizations could drop it to C being 3x faster. So still kind of unfortunate but at least I feel more confident about presenting such a result. I think compiling the code instead of interpreting made the biggest difference by far.

I'm not sure what run-time checks are disabled by this option in the Link Options, like if they are checks that could actually be helping instead.

Thanks for the tips and discussion.

 

[kylef]

Originally posted by: duragezic
I think compiling the code instead of interpreting made the biggest difference by far.

You said that you're running on an embedded platform? What JVM are you using?

Anyway, thanks for posting back your findings.

 

[degibson]

Originally posted by: Markbnj

Something like the CLR/VM will eventually be running in firmware.

I'm really against you on that -- runtimes and VMs are just too big for firmware, for one. And its not like these things don't themselves change all the time.

If there's one thing history has shown about software evolution, its that software accumulates in layers, and the old layers rarely, if ever, die. I fully think that future application layers will expose managed languages primarily, but managed languages may never be able to compete with raw C for performance-critical enterprise systems (e.g. commercial databases, big $) and performance-critical non-enterprise applications (e.g. games, also big $). Web applications, GUIs, etc., are of course all non-critical and can easily take a 10x slowdown on the client side without the user noticing. (not to say that the managed language impact would be 10x, that was just a number)