Groovy Micro Benchmark Revisited (Groovy is fast…)
Alright, no more Groovy posts for a while after this one. In fact, I hadn’t planned on posting about this again, but I thought it was worth mention.
From the last post, we saw the following results for parsing an XML dataset with about 450 simple elements in it 1,000 times:
- Java: ~9 seconds
- Groovy: ~28 seconds (with new results below…)
I decided to rerun the test for Groovy because of a comment John Wilson left on the first post. He suggested this more concise syntax for the Groovy implementation:
I reran the same benchmark, and I was very, very surprised at the results:
- Java: ~9 seconds
- Groovy:
~28~9 seconds
All I can say is wow. I’m not sure what is happening differently, but I’m happy:)
Any ideas what was taking up so much time in the original Groovy implementation?
Enjoyed this post? Click to get future articles delivered by email or get the RSS feed.
I’m really quite surprised too!
Three operations went via the MetaClass in the old version and don’t in this version (setting the name and email properties on the Contact instance and appending the Contact instance to the list).
The operations on the object produced by XmlSlurper will be quite fast because it short circuits most of the MataClass code to perform the magic (disclosure: I wrote XmlSlurper!).
Even so, it really should still be slower than Java by quite a margin.
All I can suggest is that the Java XML library you are using is quite slow and the extra performance you are getting from XmlSlurper is compensating for the slower Groovy performance.
The Java library I’m using is JDOM, which gives a level of indirection over the low level parsers like SAX, DOM, etc. I’m not sure how much of a performance hit that causes, but I’d be willing to bet that execution time would be shorter if I did a low level SAX impl myself.
All in all, I’m very pleased to see that Groovy performance is at least on par with Java performance (anecdotal benchmark notwithstanding).
Thanks again for weighing in John, and great job on XmlSlurper, especially the name:).
Unfortunately I’ve seen this happen too many times in Groovy. A slightly different implementation yields incomprehensibly different results. This is really bad by any stretch of imagination. Developers are not (and cannot be) profiling every line of code.
Sorry to burst the performance bubble here… but your second benchmark is more Java code that Groovy code. XmlSlurper is all Java…
http://svn.codehaus.org/groovy/trunk/groovy/groovy-core/src/main/groovy/util/XmlSlurper.java
So the title here is a bit misleading, Groovy is still slow. :-/ The only way to really tell for sure is to write various intensive algorithms in both languages (preferrably as free of other libraries and utilities as possible).
Something like this:
http://www.javalobby.org/java/forums/t101157.html
Not that anyone would actually write a raytracer in Groovy, but if we’re talking performance, it’s exactly the kind of thing needed to demonstrate Groovy’s performance characteristics in general. And besides, the Scala implementation somehow managed to be faster than Java….(?).
I’d like to see more benchmarks like that one.
Clinton
Clinton,
Good point about XmlSlurper (I hadn’t considered it was written in Java), but in this context I think the point is that it’s all (both the Java and Groovy) bytecode.
And you are right, the title “Groovy is fast” is misleading, especially since I said in the previous post that “There is no such thing as saying Groovy is slow.” I think I should have said “Groovy is fast enough”. Thanks for pointing this out.
I’d like to see more benchmarks like the ray tracer too, especially after the clean up of Grooy’s MOP and some optimizations are added for working with primitive types.
I forgot to mention… I don’t even really care that much about Java vs. Groovy performance. Although Java makes a good baseline for judging the performance of other languages. (e.g. consider Java == 1.0, where a language that scores 2.3 is 2.3 times slower and a language that scores 0.5 is twice as fast.)
The most important comparison I would like to see is JRuby vs. Groovy. Unfortunately the raytracer benchmark didn’t include JRuby, but I’d be very interested to see how it fairs. Jython too!
I think we need a benchmark suite for JVM languages!
Clinton
That’s a very interesting idea, Clinton. I wonder if it’s worth fiddling with that ray-tracer benchmark and porting it to different languages. Actually, this sounds a lot more like a generic OSS project to devise a standardized set of benchmarks.
i had a small experiment before:
http://www.jroller.com/ff/entry/groovy_good_bad_and_the
I am pretty excited about groovy, but the performance has me a little worried. I wrote a simple loop from 1 to 10,000,000. Nothing but a loop that just set some variable to the counter. It took 50ms in Java and 10 seconds in groovy. That is not a bit slower, that’s not even 10 times slower. That’s over 100 times slower.
I don’t think a loop to ten million is common, but if I want to parse 100,000 entries in an XML file, how can it possibly keep up?
Ian,
I wonder if the difference you saw related to how Groovy uses numbers (primitives) – since it wraps all primitives in their equivalent Java wrapper classes (int -> Integer), I’d imagine that would slow things down quite a bit. I’d be interested in seeing the performance of Java code that loops 10,000,000 using the wrapper classes instead of ints. Also, did you try running the Groovy test more than once in the same run? I wonder if the JVM would cache those generated Integers from the first run and use them the second time (isn’t this called interning with Strings?).
There’s talk that there are going to be some optimizations around the way Groovy internally handles numbers – that could speed things up quite a bit.
Groovy (like most scripting languages) is slower than many other “heavyweight” languages (like Java) – but one of the great things about Groovy is how easy it is to fall back on Java if your really need to write some performant code.
Right now I couldn’t see myself writing an entire application in Groovy – I’m sure there would be parts where Java would be the better fit. Thus the wonder of Groovy:)
What was the code you used for your test?
For me, at least, benchmarking Groovy vs JRuby vs Jython, etc, isn’t really the point. The overwhelming advantage of Groovy for me is that the learning curve is so smooth and easy. I also love that if I don’t know how to do something in Groovy, I just drop back to Java and do it that way with no loss whatsoever. I can always go back and make my code groovier.
Speed comes with maturity. Java was dog slow in the early days, but the productivity gains it gave me vs C++ were worth it. If I have any XML to process at all, picking Groovy over Java is a no brainer. If the resulting code turns out to be too slow, I can always drop back to a Java implementation later.
I’m sure that each new version of Groovy will by much faster than the last one. In the meantime, writing Groovy is just so natural and easy that coding is fun again.
Ken,
That is exactly why I really got excited about groovy! The language is really nice and I can fall back to Java when need be. I would use it for my own scripts, plug-ins, etc… but I don’t think I would want to roll out a new app with it yet.
Jonchase,
About the wrapping of the ints, I assumed this was the reason too. My code really was just a for loop, exactly the same code in Java and groovy. I tried to used “Integers” in java too, and nothing changed, although I’d chalk that up to clever Java optimizations.
Ian,
Being able to fall back to Java is what really interested me in Groovy in the first place. I’ve always wanted to try Ruby and Rails, but I could never make the leap – there was just too much I didn’t know. Groovy makes things nice and comfortable – familiar syntax and being able to reuse all of my Java knowledge where big factors in deciding to learn Groovy.
Ian,
If you benchmark other currently-popular dynamic languages like Ruby and JRuby, I think you’ll find their execution times are around the same. JRuby actually seems to be slower on pure loops than normal Ruby.
No matter what dynamic “scripting” language you use, I think its probably best not to use it to write loops with 10 million iterations.
My results for a 1..10m loop were:
Java: .042s
Ruby 1.8.6: 4.407s
Groovy: 6.407s
JRuby: 11.391s
JRuby 1.1 RC2: 11.407
Ruby 1.9.0: 10.344 (with some weird behavior)
Is there anything happening inside your Java loop, like incrementing a variable declared outside the scope of the loop? I wouldn’t be surprised if javac is smart enough to remove the loop when compiling if it sees it’s not doing anything.
Still, looks like you are right, Grant. Scripting languages don’t seem to be as fast (for a 10m loop at least). I think the general consensus in the comments is that they’re fast enough though.
How crazy is it that we’re comparing language performance with JAVA as a baseline? A few years back (and even currently, though much less so) it was “Java isn’t as fast as C++”. It’s amazing that Java has come so far and gotten so “fast”.
And even better, the scripting languages, especially Groovy, will keep getting faster too, which is great news for Java.
jonchase,
The only thing inside my loop is:
if(i % 100000 == 0) System.out.print(’.');
Java prints this out as expected. Interestingly, Ruby 1.9.0 (which is basically 2.0 RC1) does not. It pauses for a while (10 seconds), then spits out all the dots at once. I’m guessing this is a bug in its bytecode compiler, which is new to 1.9.
As I understand it though, Java has been a match for C/C++ in many areas since it got a good JIT compiler. Frankly I hated Java before then; its performance was terrible.
I’d imagine Java is doing some optimizations (at runtime, I’m told javac’s -O option is now useless), but its not smart enough to sidestep the useless loop and just print some dots. Increasing the iterations to 100 million and the mod value to 1 million (i.e., keeping the number of dots printed the same, but increasing the loop length by 10) produces a delay of .828 seconds.
I’m late to the party, but there are probably many factors influencing Grant’s JRuby results. For example, JRuby takes longer to warm up than normal Ruby because of its Ruby JIT and then the JVM’s JIT. So the benchmark is a bit too short. If it’s run more than once, numbers look much better. Here’s an example:
Script:
1..10_000_000.times {|i| print ‘.’ if i % 100_000 == 0}
JRuby RC3: 5.5s
JRuby RC3 looping five times: 18.8s
Ruby 1.8: 5s
Ruby 1.8 looping five times: 25.2s
It’s also important to run on the most current JVM possible, since that has a tremendous impact on performance.
Charles,
Interesting results. I’m absolutely amazed and impressed at how far JRuby has come. Keep it up:).
You may also want to look at vtd-xml,the latest and most advanced xml technology
vtd-xml